{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for Happy Customer Bank"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.调整树的参数：subsample 和 colsample_bytree\n",
    "(粗调，参数的步长为0.1；下一步是在粗调最佳参数周围，将步长降为0.05，进行精细调整)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据说明:\n",
    "数据集共26个字段: 其中1-24列为输入特征，25-26列为输出特征。\n",
    "    1. ID - 唯一ID（不能用于预测）\n",
    "    2. Gender - 性别\n",
    "    3. City - 城市\n",
    "    4. Monthly_Income - 月收入（以卢比为单位）\n",
    "    5. DOB - 出生日期\n",
    "    6. Lead_Creation_Date - 潜在（贷款）创建日期\n",
    "    7. Loan_Amount_Applied - 贷款申请请求金额（印度卢比，INR）\n",
    "    8. Loan_Tenure_Applied - 贷款申请期限（单位为年）\n",
    "    9. Existing_EMI -现有贷款的EMI（EMI：电子货币机构许可证） \n",
    "    10. Employer_Name雇主名称\n",
    "    11. Salary_Account - 薪资帐户银行\n",
    "    12. Mobile_Verified - 是否移动验证（Y / N）\n",
    "    13. VAR5 - 连续型变量\n",
    "    14. VAR1-  类别型变量\n",
    "    15. Loan_Amount_Submitted - 提交的贷款金额（在看到资格后修改和选择）\n",
    "    16. Loan_Tenure_Submitted - 提交的贷款期限（单位为年，在看到资格后修改和选择）\n",
    "    17. Interest_Rate - 提交贷款金额的利率\n",
    "    18. Processing_Fee - 提交贷款的处理费（INR）\n",
    "    19. EMI_Loan_Submitted -提交的EMI贷款金额（INR）\n",
    "    20. Filled_Form - 后期报价后是否已填写申请表格\n",
    "    21. Device_Type - 进行申请的设备（浏览器/移动设备）\n",
    "    22. Var2 - 类别型变量\n",
    "    23. Source - 类别型变量\n",
    "    24. Var4 - 类别型变量\n",
    "\n",
    "输出：\n",
    "    25. LoggedIn - 是否login（只用于理解问题的变量，不能用于预测，测试集中没有）\n",
    "    26. Disbursed - 是否发放贷款（目标变量），1为发放贷款（目标客户）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.read_csv('train_save.csv')\n",
    "test = pd.read_csv('test_save.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((87020, 52), (37717, 51))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.shape, test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEKCAYAAADaa8itAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFpFJREFUeJzt3Xu0JWV95vHvw02Qi4C2LtNAGmNHJTreWoImEy8oAk5oZEGCyxEGmcFkAHESHTHJsuNdEoRI4iWsQLiMw0UGtEWUQQSdOIo0F8UGWXQwQg8EWkFEGGEafvPHfg/s7j6Xorv2OWzP97PWXqfqrbdq/7ar5VlVb9VbqSokSerDZnNdgCTpV4ehIknqjaEiSeqNoSJJ6o2hIknqjaEiSeqNoSJJ6o2hIknqjaEiSerNFnNdwGx7xjOeUYsWLZrrMiRpbFxzzTU/qaoFXfrOu1BZtGgRK1asmOsyJGlsJPlx175e/pIk9cZQkST1xlCRJPXGUJEk9cZQkST1xlCRJPXGUJEk9cZQkST1xlCRJPVm3j1Rv6le/p6z5roEPQld89eHzXUJ0pOCZyqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTejDRUkvyXJCuT/CDJOUm2TrJ7kquS3JLkvCRbtb5Paeur2vZFQ8d5X2u/Ockbh9r3bW2rkhw/yt8iSZrZyEIlyULgncCSqnohsDlwKHACcHJVLQbuBY5suxwJ3FtVzwVObv1Iskfb77eAfYFPJ9k8yebAp4D9gD2At7S+kqQ5MurLX1sA2yTZAngqcCfwOuCCtv1M4MC2vLSt07bvnSSt/dyqeqiqfgSsAvZsn1VVdWtVPQyc2/pKkubIyEKlqv4PcCJwG4MwuQ+4BvhZVa1t3VYDC9vyQuD2tu/a1v/pw+3r7TNVuyRpjozy8tdODM4cdgd+DdiWwaWq9dXELlNse6Ltk9VyVJIVSVasWbNmptIlSRtplJe/Xg/8qKrWVNX/Ay4EXgXs2C6HAewC3NGWVwO7ArTtTwPuGW5fb5+p2jdQVadW1ZKqWrJgwYI+fpskaRKjDJXbgL2SPLWNjewN3AhcARzc+hwOfLEtL2/rtO1fr6pq7Ye2u8N2BxYD3wWuBha3u8m2YjCYv3yEv0eSNIORvaSrqq5KcgFwLbAWuA44FfgycG6SD7e209oupwFnJ1nF4Azl0HaclUnOZxBIa4Gjq+oRgCTHAJcyuLPs9KpaOarfI0ma2Ujf/FhVy4Bl6zXfyuDOrfX7/hI4ZIrjfAT4yCTtlwCXbHqlkqQ++ES9JKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3M4ZKkm2TbNaWfzPJAUm2HH1pkqRx0+VM5ZvA1kkWApcDRwBnjLIoSdJ46hIqqaoHgYOAv62qNwN7jLYsSdI46hQqSV4JvBX4cmvbYnQlSZLGVZdQeRfwPuCiqlqZ5DnAFaMtS5I0jmY846iqbwDfSLJtW78VeOeoC5MkjZ8ud3+9MsmNwE1t/cVJPj3yyiRJY6fL5a+/Ad4I/BSgqr4H/N4oi5IkjadODz9W1e3rNT0yglokSWOuy11ctyd5FVBJtmIwnnLTaMuSJI2jLmcqfwQcDSwEVgMvaeuSJK2jy91fP2HwjIokSdPqcvfXmUl2HFrfKcnpoy1LkjSOulz++jdV9bOJlaq6F3jp6EqSJI2rLqGyWZKdJlaS7IzTtEiSJtElHD4B/O8kF7T1Q4CPjK4kSdK46jJQf1aSa4DXAgEOqqobR16ZJGnsdH3z4w+BC4EvAr9IsluXnZLsmOSCJD9MclOb8mXnJJcluaX93an1TZJTkqxK8v0kLxs6zuGt/y1JDh9qf3mSG9o+pyRJ958uSepbl7u/jgXuAi4DLmYw/f3FHY//SeCrVfV84MUMHpo8Hri8qhYzeOnX8a3vfsDi9jkK+Ez7/p2BZcBvA3sCy4bGeD7T+k7st2/HuiRJI9BlTOU44HlV9dMncuAkOzCYI+w/AFTVw8DDSZYCr2ndzgSuBN4LLAXOqqoCvtPOcp7d+l5WVfe0414G7JvkSmCHqvp2az8LOBD4yhOpU5LUny6Xv24H7tuIYz8HWAP8Y5LrkvxDmz7/WVV1J0D7+8zWf2H7rgmrW9t07asnaZckzZEuZyq3Alcm+TLw0ERjVZ3U4dgvA46tqquSfJLHL3VNZrLxkNqI9g0PnBzF4DIZu+3WaThIkrQRupyp3MZgPGUrYPuhz0xWA6ur6qq2fgGDkLmrXdai/b17qP+uQ/vvAtwxQ/suk7RvoKpOraolVbVkwYIFHUqXJG2MLrcUfwAgybZV9UDXA1fVvya5PcnzqupmYG/gxvY5HPh4+/vFtsty4Jgk5zIYlL+vqu5Mcinw0aHB+X2A91XVPUnuT7IXcBVwGPC3XeuTJPVvxlBJ8krgNGA7YLckLwbeUVX/ucPxjwU+16bMvxU4gsHZ0flJjmRwFnRI63sJsD+wCniw9aWFx4eAq1u/D04M2gN/DJwBbMNggN5BekmaQ13GVCbe/LgcBm9+TNLpzY9VdT2wZJJNe0/St5hiSv2qOh3YYBLLqloBvLBLLZKk0fPNj5Kk3vjmR0lSb3zzoySpN9OeqSTZHHhbVfnmR0nSjKY9U6mqRxhMnyJJ0oy6jKl8K8nfAecBjz2nUlXXjqwqSdJY6hIqr2p/PzjUVsDr+i9HkjTOZhpT2Qz4TFWdP0v1SJLG2ExjKo8Cx8xSLZKkMdflluLLkrw7ya7trY07txdnSZK0ji5jKm9vf4efTSkG70uRJOkxXWYp3n02CpEkjb8usxQfNll7VZ3VfzmSpHHW5fLXK4aWt2Yww/C1gKEiSVpHl8tfxw6vJ3kacPbIKpIkja1OU9+v50Fgcd+FSJLGX5cxlS8xuNsLBiG0B+DDkJKkDXQZUzlxaHkt8OOqWj2ieiRJY6xLqNwG3FlVvwRIsk2SRVX1LyOtTJI0drqMqXweeHRo/ZHWJknSOrqEyhZV9fDESlveanQlSZLGVZdQWZPkgImVJEuBn4yuJEnSuOoypvJHwOfai7pg8J76SZ+ylyTNb10efvxnYK8k2wGpqvtHX5YkaRzNePkryUeT7FhVv6iq+5PslOTDs1GcJGm8dBlT2a+qfjaxUlX3AvuPriRJ0rjqEiqbJ3nKxEqSbYCnTNNfkjRPdRmo/2/A5Un+kcF0LW8HzhxpVZKksdRloP6vknwfeH1r+lBVXTrasiRJ46jLmQrAdcCWDM5UrhtdOZKkcdbl7q8/AL4LHAz8AXBVkoNHXZgkafx0OVP5c+AVVXU3QJIFwNeAC0ZZmCRp/HS5+2uziUBpftpxP0nSPNPlTOWrSS4FzmnrfwhcMrqSJEnjqsvdX+9JchDwu0CAU6vqopFXJkkaO53u/qqqC4ELR1yLJGnMOTYiSeqNoSJJ6s2UoZLk8vb3hNkrR5I0zqYbU3l2klcDByQ5l8Eg/WOq6tqRViZJGjvTXf56P3A8sAtwEvCJoc+JXb8gyeZJrktycVvfPclVSW5Jcl6SrVr7U9r6qrZ90dAx3tfab07yxqH2fVvbqiTHd//ZkqRRmDJUquqCqtoP+Kuqeu16n9c9ge84DrhpaP0E4OSqWgzcCxzZ2o8E7q2q5wInt34k2QM4FPgtYF/g0y2oNgc+BewH7AG8pfWVJM2RGQfqq+pDSQ5IcmL7/LuuB0+yC/Am4B/aeoDX8fgUL2cCB7blpTw+pf4FwN6t/1Lg3Kp6qKp+BKwC9myfVVV1a1U9DJzb+kqS5kiXCSU/xuBs48b2Oa61dfE3wH8FHm3rTwd+VlVr2/pqYGFbXgjcDtC239f6P9a+3j5TtU/2G45KsiLJijVr1nQsXZL0RHW5pfhNwBuq6vSqOp3BJag3zbRTO6O5u6quGW6epGvNsO2Jtm/YWHVqVS2pqiULFiyYpmpJ0qbo+j6VHYF72vLTOu7zOwzuHNsf2BrYgcGZy45JtmhnI7sAd7T+q4FdgdVJtmjfc89Q+4ThfaZqlyTNgS5nKh8DrktyRpIzgWuAj860U1W9r6p2qapFDAbav15VbwWuYPBuFoDDgS+25eVtnbb961VVrf3QdnfY7sBiBu93uRpY3O4m26p9x/IOv0eSNCJdJpQ8J8mVwCsYXHJ6b1X96yZ853uBc5N8mMFbJE9r7acBZydZxeAM5dD2/SuTnM9gPGctcHRVPQKQ5BjgUmBz4PSqWrkJdUmSNlHXCSXvZBPOAqrqSuDKtnwrgzu31u/zS+CQKfb/CPCRSdovwWn4JelJw7m/JEm9MVQkSb2ZNlSSbJbkB7NVjCRpvE0bKlX1KPC9JLvNUj2SpDHWZaD+2cDKJN8FHphorKoDRlaVJGksdQmVD4y8CknSr4Quz6l8I8mvA4ur6mtJnsrguRBJktbRZULJ/8Rg1uC/b00LgS+MsihJ0njqckvx0Qzm8fo5QFXdAjxzlEVJksZTl1B5qL2vBIA22eOkswFLkua3LqHyjSR/BmyT5A3A54EvjbYsSdI46hIqxwNrgBuAdzCYa+svRlmUJGk8dbn769E25f1VDC573dympJckaR0zhkqSNwGfBf6ZwdT3uyd5R1V9ZdTFSZLGS5eHHz8BvLaqVgEk+Q3gy4ChIklaR5cxlbsnAqW5Fbh7RPVIksbYlGcqSQ5qiyuTXAKcz2BM5RAGr/KVJGkd013++v2h5buAV7flNcBOI6tIkjS2pgyVqjpiNguRJI2/Lnd/7Q4cCywa7u/U95Kk9XW5++sLwGkMnqJ/dLTlSJLGWZdQ+WVVnTLySiRJY69LqHwyyTLgfwIPTTRW1bUjq0qSNJa6hMqLgLcBr+Pxy1/V1iVJekyXUHkz8Jzh6e8lSZpMlyfqvwfsOOpCJEnjr8uZyrOAHya5mnXHVLylWJK0ji6hsmzkVUiSfiV0eZ/KN2ajEEnS+OvyRP39PP5O+q2ALYEHqmqHURYmSRo/Xc5Uth9eT3IgsOfIKpIkja0ud3+to6q+gM+oSJIm0eXy10FDq5sBS3j8cpgkSY/pcvfX8HtV1gL/AiwdSTWSpLHWZUzF96pIkjqZ7nXC759mv6qqD42gHknSGJvuTOWBSdq2BY4Eng4YKpKkdUz3OuFPTCwn2R44DjgCOBf4xFT7SZLmr2nHVJLsDPwJ8FbgTOBlVXXvbBQmSRo/Uz6nkuSvgauB+4EXVdVfPpFASbJrkiuS3JRkZZLjWvvOSS5Lckv7u1NrT5JTkqxK8v0kLxs61uGt/y1JDh9qf3mSG9o+pyTJRvxvIEnqyXQPP/4p8GvAXwB3JPl5+9yf5Ocdjr0W+NOqegGwF3B0kj2A44HLq2oxcHlbB9gPWNw+RwGfgcfOlpYBv83gSf5lE0HU+hw1tN++3X62JGkUpgyVqtqsqrapqu2raoehz/Zd5v2qqjsnXjlcVfcDNwELGTzjcmbrdiZwYFteCpxVA98BdkzybOCNwGVVdU87U7oM2Ldt26Gqvl1VBZw1dCxJ0hx4wtO0bIwki4CXAlcBz6qqO2EQPMAzW7eFwO1Du61ubdO1r56kfbLvPyrJiiQr1qxZs6k/R5I0hZGHSpLtgP8BvKuqprtsNtl4SG1E+4aNVadW1ZKqWrJgwYKZSpYkbaSRhkqSLRkEyueq6sLWfFe7dEX7e3drXw3sOrT7LsAdM7TvMkm7JGmOjCxU2p1YpwE3VdVJQ5uWAxN3cB0OfHGo/bB2F9hewH3t8tilwD5JdmoD9PsAl7Zt9yfZq33XYUPHkiTNgS4TSm6s3wHeBtyQ5PrW9mfAx4HzkxwJ3AYc0rZdAuwPrAIeZPCgJVV1T5IPMbi9GeCDVXVPW/5j4AxgG+Ar7SNJmiMjC5Wq+icmH/cA2HuS/gUcPcWxTgdOn6R9BfDCTShTktSjWbn7S5I0PxgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN6Mfagk2TfJzUlWJTl+ruuRpPlsi7kuYFMk2Rz4FPAGYDVwdZLlVXXj3FYmzY3bPviiuS5BT0K7vf+GWfuucT9T2RNYVVW3VtXDwLnA0jmuSZLmrXEPlYXA7UPrq1ubJGkOjPXlLyCTtNUGnZKjgKPa6i+S3DzSquaPZwA/mesingxy4uFzXYI25L/PCcsm+0/lE/LrXTuOe6isBnYdWt8FuGP9TlV1KnDqbBU1XyRZUVVL5roOaTL++5wb437562pgcZLdk2wFHAosn+OaJGneGuszlapam+QY4FJgc+D0qlo5x2VJ0rw11qECUFWXAJfMdR3zlJcU9WTmv885kKoNxrUlSdoo4z6mIkl6EjFUNKOZpsJJ8pQk57XtVyVZNPtVaj5KcnqSu5P8YIrtSXJK+7f5/SQvm+0a5xtDRdMamgpnP2AP4C1J9liv25HAvVX1XOBk4ITZrVLz2BnAvtNs3w9Y3D5HAZ+ZhZrmNUNFM+kyFc5S4My2fAGwd5JNftpKmklVfRO4Z5ouS4GzauA7wI5Jnj071c1Phopm0mUqnMf6VNVa4D7g6bNSnTQ9p3KaZYaKZtJlKpxO0+VIc8B/m7PMUNFMukyF81ifJFsAT2P6SxLSbOk0lZP6Y6hoJl2mwlkOTMyoeDDw9fIBKD05LAcOa3eB7QXcV1V3znVRv8rG/ol6jdZUU+Ek+SCwoqqWA6cBZydZxeAM5dC5q1jzSZJzgNcAz0iyGlgGbAlQVZ9lMNvG/sAq4EHgiLmpdP7wiXpJUm+8/CVJ6o2hIknqjaEiSeqNoSJJ6o2hIknqjaEidZTkkSTXJ1mZ5HtJ/iTJZm3bkiSnTLPva5JcPHvVbvD9f5nk3XP1/Zo/fE5F6u7/VtVLAJI8E/jvDGYPWFZVK4AVo/riJFu0edWkJzXPVKSNUFV3M5hK/Zj2tPZjZyJJXt3OaK5Pcl2S7dtuOyS5KMmNST47dJbzi4njJjk4yRlt+YwkJyW5AjhhquMmeU+Sq9v7Qj4wdKw/b+/B+RrwvNn430XyTEXaSFV1awuGZ6636d3A0VX1rSTbAb9s7XsyeCfNj4GvAgcxeFXAdH4TeH1VPZLkS+sfN8k+DN4VsieDyROXJ/k94AEGMxu8lMH/z68Frtm0XyzNzDMVadNMNgvut4CTkrwT2HHostV323tpHgHOAX63w/E/3/pPddx92uc6BsHxfAYh82+Bi6rqwar6ORvO1yaNhKEibaQkzwEeAe4ebq+qjwP/EdgG+E6S509sWu8QNUn71uv1eWCG4wb4WFW9pH2eW1WnTfF90sgZKtJGSLIA+Czwd+vPyJzkN6rqhqo6gcHg/USo7Nlme94M+EPgn1r7XUle0NrfPM13TnbcS4G3t8thJFnYbiL4JvDmJNu0sZff7+u3S9NxTEXqbpsk1zOYBXctcDZw0iT93pXktQzOYm4EvgK8Evg28HHgRQz+o39R6388cDGDNxT+ANhuiu/f4LhV9VCSFwDfbm9w/gXw76vq2iTnAdczGMP5X5vyw6WunKVYktQbL39JknpjqEiSemOoSJJ6Y6hIknpjqEiSemOoSJJ6Y6hIknpjqEiSevP/AQfaa5oYJqxDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(train.Disbursed);\n",
    "pyplot.xlabel('Disbursed');\n",
    "pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "样本不均衡，使用交叉验证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "默认参数，此时学习率为0.1，比较大，观察弱分类数目的大致范围\n",
    "（采用默认参数配置，看看模型是过拟合还是欠拟合）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = train['Disbursed']\n",
    "train = train.drop([\"Disbursed\"], axis=1)\n",
    "X_train = np.array(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 87020 entries, 0 to 87019\n",
      "Data columns (total 51 columns):\n",
      "Existing_EMI                     87020 non-null float64\n",
      "Loan_Amount_Applied              87020 non-null float64\n",
      "Loan_Tenure_Applied              87020 non-null float64\n",
      "Monthly_Income                   87020 non-null int64\n",
      "Var4                             87020 non-null int64\n",
      "Var5                             87020 non-null int64\n",
      "Age                              87020 non-null int64\n",
      "EMI_Loan_Submitted_Missing       87020 non-null int64\n",
      "Interest_Rate_Missing            87020 non-null int64\n",
      "Loan_Amount_Submitted_Missing    87020 non-null int64\n",
      "Loan_Tenure_Submitted_Missing    87020 non-null int64\n",
      "Processing_Fee_Missing           87020 non-null int64\n",
      "Device_Type_0                    87020 non-null int64\n",
      "Device_Type_1                    87020 non-null int64\n",
      "Filled_Form_0                    87020 non-null int64\n",
      "Filled_Form_1                    87020 non-null int64\n",
      "Gender_0                         87020 non-null int64\n",
      "Gender_1                         87020 non-null int64\n",
      "Var1_0                           87020 non-null int64\n",
      "Var1_1                           87020 non-null int64\n",
      "Var1_2                           87020 non-null int64\n",
      "Var1_3                           87020 non-null int64\n",
      "Var1_4                           87020 non-null int64\n",
      "Var1_5                           87020 non-null int64\n",
      "Var1_6                           87020 non-null int64\n",
      "Var1_7                           87020 non-null int64\n",
      "Var1_8                           87020 non-null int64\n",
      "Var1_9                           87020 non-null int64\n",
      "Var1_10                          87020 non-null int64\n",
      "Var1_11                          87020 non-null int64\n",
      "Var1_12                          87020 non-null int64\n",
      "Var1_13                          87020 non-null int64\n",
      "Var1_14                          87020 non-null int64\n",
      "Var1_15                          87020 non-null int64\n",
      "Var1_16                          87020 non-null int64\n",
      "Var1_17                          87020 non-null int64\n",
      "Var1_18                          87020 non-null int64\n",
      "Var2_0                           87020 non-null int64\n",
      "Var2_1                           87020 non-null int64\n",
      "Var2_2                           87020 non-null int64\n",
      "Var2_3                           87020 non-null int64\n",
      "Var2_4                           87020 non-null int64\n",
      "Var2_5                           87020 non-null int64\n",
      "Var2_6                           87020 non-null int64\n",
      "Mobile_Verified_0                87020 non-null int64\n",
      "Mobile_Verified_1                87020 non-null int64\n",
      "Source_0                         87020 non-null int64\n",
      "Source_1                         87020 non-null int64\n",
      "Source_2                         87020 non-null int64\n",
      "Source_3                         87020 non-null int64\n",
      "Source_4                         87020 non-null int64\n",
      "dtypes: float64(3), int64(48)\n",
      "memory usage: 33.9 MB\n"
     ]
    }
   ],
   "source": [
    "train.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第二轮参数调整得到的n_estimators最优值（86），其余参数继续默认值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'colsample_bytree': [0.6, 0.7, 0.8, 0.9],\n",
       " 'subsample': [0.3, 0.4, 0.5, 0.6, 0.7, 0.8]}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#max_depth 建议3-10， min_child_weight=1／sqrt(ratio_rare_event) =5.5\n",
    "subsample = [i/10.0 for i in range(3,9)]\n",
    "colsample_bytree = [i/10.0 for i in range(6,10)]\n",
    "param_test3_1 = dict(subsample=subsample, colsample_bytree=colsample_bytree)\n",
    "param_test3_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/model_selection/_search.py:762: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.06445, std: 0.00111, params: {'subsample': 0.3, 'colsample_bytree': 0.6},\n",
       "  mean: -0.06412, std: 0.00105, params: {'subsample': 0.4, 'colsample_bytree': 0.6},\n",
       "  mean: -0.06399, std: 0.00121, params: {'subsample': 0.5, 'colsample_bytree': 0.6},\n",
       "  mean: -0.06386, std: 0.00108, params: {'subsample': 0.6, 'colsample_bytree': 0.6},\n",
       "  mean: -0.06380, std: 0.00114, params: {'subsample': 0.7, 'colsample_bytree': 0.6},\n",
       "  mean: -0.06381, std: 0.00121, params: {'subsample': 0.8, 'colsample_bytree': 0.6},\n",
       "  mean: -0.06405, std: 0.00112, params: {'subsample': 0.3, 'colsample_bytree': 0.7},\n",
       "  mean: -0.06402, std: 0.00102, params: {'subsample': 0.4, 'colsample_bytree': 0.7},\n",
       "  mean: -0.06389, std: 0.00112, params: {'subsample': 0.5, 'colsample_bytree': 0.7},\n",
       "  mean: -0.06382, std: 0.00101, params: {'subsample': 0.6, 'colsample_bytree': 0.7},\n",
       "  mean: -0.06382, std: 0.00103, params: {'subsample': 0.7, 'colsample_bytree': 0.7},\n",
       "  mean: -0.06392, std: 0.00103, params: {'subsample': 0.8, 'colsample_bytree': 0.7},\n",
       "  mean: -0.06409, std: 0.00109, params: {'subsample': 0.3, 'colsample_bytree': 0.8},\n",
       "  mean: -0.06384, std: 0.00119, params: {'subsample': 0.4, 'colsample_bytree': 0.8},\n",
       "  mean: -0.06391, std: 0.00097, params: {'subsample': 0.5, 'colsample_bytree': 0.8},\n",
       "  mean: -0.06379, std: 0.00123, params: {'subsample': 0.6, 'colsample_bytree': 0.8},\n",
       "  mean: -0.06375, std: 0.00122, params: {'subsample': 0.7, 'colsample_bytree': 0.8},\n",
       "  mean: -0.06364, std: 0.00116, params: {'subsample': 0.8, 'colsample_bytree': 0.8},\n",
       "  mean: -0.06396, std: 0.00116, params: {'subsample': 0.3, 'colsample_bytree': 0.9},\n",
       "  mean: -0.06393, std: 0.00119, params: {'subsample': 0.4, 'colsample_bytree': 0.9},\n",
       "  mean: -0.06384, std: 0.00106, params: {'subsample': 0.5, 'colsample_bytree': 0.9},\n",
       "  mean: -0.06371, std: 0.00118, params: {'subsample': 0.6, 'colsample_bytree': 0.9},\n",
       "  mean: -0.06372, std: 0.00101, params: {'subsample': 0.7, 'colsample_bytree': 0.9},\n",
       "  mean: -0.06371, std: 0.00102, params: {'subsample': 0.8, 'colsample_bytree': 0.9}],\n",
       " {'colsample_bytree': 0.8, 'subsample': 0.8},\n",
       " -0.06363784022623405)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb3_1 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=86,  #第二轮参数调整得到的n_estimators最优值\n",
    "        max_depth=5,\n",
    "        min_child_weight=2,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch3_1 = GridSearchCV(xgb3_1, param_grid = param_test3_1, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch3_1.fit(X_train , y_train)\n",
    "\n",
    "gsearch3_1.grid_scores_, gsearch3_1.best_params_,     gsearch3_1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([ 6.63458185,  7.93297515,  8.60610156,  8.78002481,  8.58050804,\n",
       "         8.03869777,  7.83526559,  9.3206882 ,  9.62875471, 10.12068896,\n",
       "         9.94534082,  8.89039001,  8.84350753, 10.03181705, 10.62122774,\n",
       "        10.80552497, 10.68363166,  9.99766374,  9.57379742, 10.89819779,\n",
       "        11.60587535, 11.5109642 ,  9.06892123,  6.339959  ]),\n",
       " 'mean_score_time': array([0.07107415, 0.08702421, 0.07393298, 0.07292094, 0.07513561,\n",
       "        0.07361217, 0.07988281, 0.073313  , 0.07175269, 0.07442703,\n",
       "        0.07785416, 0.07221823, 0.0760778 , 0.06952958, 0.0698524 ,\n",
       "        0.07097239, 0.0727396 , 0.07039084, 0.07764077, 0.06918402,\n",
       "        0.06869144, 0.06196661, 0.05212379, 0.04788818]),\n",
       " 'mean_test_score': array([-0.06445407, -0.06412392, -0.0639867 , -0.0638646 , -0.06380287,\n",
       "        -0.06381183, -0.06405348, -0.06402294, -0.06388624, -0.06381866,\n",
       "        -0.06381984, -0.06391541, -0.06408916, -0.06384161, -0.0639064 ,\n",
       "        -0.06378674, -0.06375156, -0.06363784, -0.06396058, -0.06393331,\n",
       "        -0.06383577, -0.06371412, -0.06371974, -0.06370565]),\n",
       " 'mean_train_score': array([-0.0601602 , -0.05998179, -0.05965651, -0.05962459, -0.05937521,\n",
       "        -0.05940453, -0.05994692, -0.05968659, -0.05957311, -0.05945048,\n",
       "        -0.05930907, -0.05941438, -0.05980205, -0.0594219 , -0.05921454,\n",
       "        -0.05916444, -0.05918688, -0.05918879, -0.05957102, -0.05926938,\n",
       "        -0.05904284, -0.05912045, -0.05907748, -0.0591879 ]),\n",
       " 'param_colsample_bytree': masked_array(data=[0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.7, 0.7, 0.7, 0.7, 0.7,\n",
       "                    0.7, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.9, 0.9, 0.9, 0.9,\n",
       "                    0.9, 0.9],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_subsample': masked_array(data=[0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.3, 0.4, 0.5, 0.6, 0.7,\n",
       "                    0.8, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.3, 0.4, 0.5, 0.6,\n",
       "                    0.7, 0.8],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'colsample_bytree': 0.6, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.8}],\n",
       " 'rank_test_score': array([24, 23, 19, 13,  7,  8, 21, 20, 14,  9, 10, 16, 22, 12, 15,  6,  5,\n",
       "         1, 18, 17, 11,  3,  4,  2], dtype=int32),\n",
       " 'split0_test_score': array([-0.06415763, -0.06325949, -0.06339434, -0.06311752, -0.06324988,\n",
       "        -0.06325567, -0.06318906, -0.06334777, -0.06303484, -0.06322741,\n",
       "        -0.06292601, -0.06347899, -0.06335106, -0.06283524, -0.06351736,\n",
       "        -0.06300703, -0.06285514, -0.0630445 , -0.06330414, -0.0632262 ,\n",
       "        -0.06334534, -0.06275506, -0.06300983, -0.06296311]),\n",
       " 'split0_train_score': array([-0.06058449, -0.05996013, -0.05985963, -0.05958384, -0.05945552,\n",
       "        -0.05933665, -0.06026425, -0.06009295, -0.05965556, -0.05986917,\n",
       "        -0.05947517, -0.06011132, -0.05999309, -0.05965895, -0.05944019,\n",
       "        -0.05934958, -0.05927272, -0.05942249, -0.0598175 , -0.0594496 ,\n",
       "        -0.05936651, -0.05913375, -0.05915875, -0.0591873 ]),\n",
       " 'split1_test_score': array([-0.06606602, -0.06558763, -0.06571666, -0.06539611, -0.0652983 ,\n",
       "        -0.0655868 , -0.06570499, -0.06560817, -0.06521037, -0.06511949,\n",
       "        -0.06531521, -0.06527831, -0.06561898, -0.06551403, -0.06512989,\n",
       "        -0.06557292, -0.06546448, -0.06492928, -0.06580367, -0.0657049 ,\n",
       "        -0.06525794, -0.06520732, -0.06513655, -0.06504841]),\n",
       " 'split1_train_score': array([-0.05972008, -0.05982404, -0.05944297, -0.05922975, -0.05902973,\n",
       "        -0.05923548, -0.05966248, -0.05926869, -0.05914111, -0.05890021,\n",
       "        -0.05891215, -0.05886015, -0.05970252, -0.05900653, -0.05898979,\n",
       "        -0.05871884, -0.05869171, -0.05868261, -0.05896357, -0.05881001,\n",
       "        -0.05851165, -0.0586738 , -0.05847461, -0.05855746]),\n",
       " 'split2_test_score': array([-0.06534204, -0.06519988, -0.06513679, -0.06492785, -0.06506075,\n",
       "        -0.06491048, -0.06503394, -0.06484108, -0.06528779, -0.06495788,\n",
       "        -0.06481111, -0.06502112, -0.06517649, -0.0650024 , -0.06499231,\n",
       "        -0.06496933, -0.06497983, -0.06515425, -0.06481705, -0.06501065,\n",
       "        -0.06493915, -0.06512023, -0.06473899, -0.06485705]),\n",
       " 'split2_train_score': array([-0.05992472, -0.05952785, -0.05950134, -0.05918768, -0.05912953,\n",
       "        -0.0590616 , -0.05974889, -0.05903891, -0.05933376, -0.05917009,\n",
       "        -0.05907777, -0.05924866, -0.0594264 , -0.05910609, -0.05897873,\n",
       "        -0.05902745, -0.05883293, -0.05900075, -0.05938101, -0.059193  ,\n",
       "        -0.05885222, -0.05900848, -0.05888712, -0.05904968]),\n",
       " 'split3_test_score': array([-0.06367289, -0.06343808, -0.0628886 , -0.06315453, -0.06272908,\n",
       "        -0.06262162, -0.06350714, -0.06326875, -0.06307028, -0.06277275,\n",
       "        -0.06305669, -0.06282609, -0.06342513, -0.06325413, -0.06300667,\n",
       "        -0.06262388, -0.06293509, -0.06257737, -0.06281738, -0.06284175,\n",
       "        -0.06291504, -0.06280784, -0.06294906, -0.06274307]),\n",
       " 'split3_train_score': array([-0.06027766, -0.06029204, -0.0597706 , -0.06028692, -0.05964514,\n",
       "        -0.05952702, -0.06008801, -0.05995013, -0.05988561, -0.05968918,\n",
       "        -0.05946595, -0.05938506, -0.05991109, -0.05963711, -0.05914538,\n",
       "        -0.05932302, -0.0594804 , -0.0592755 , -0.05989469, -0.05925658,\n",
       "        -0.05901951, -0.05918041, -0.0595741 , -0.05950654]),\n",
       " 'split4_test_score': array([-0.06303159, -0.06313439, -0.06279693, -0.06272687, -0.06267617,\n",
       "        -0.06268438, -0.06283215, -0.0630488 , -0.06282776, -0.06301565,\n",
       "        -0.06299004, -0.06297234, -0.062874  , -0.06260213, -0.06288564,\n",
       "        -0.06276036, -0.06252307, -0.06248363, -0.06306047, -0.06288288,\n",
       "        -0.06272122, -0.06268003, -0.06276411, -0.06291647]),\n",
       " 'split4_train_score': array([-0.06029404, -0.0603049 , -0.05970799, -0.05983478, -0.05961613,\n",
       "        -0.05986188, -0.05997097, -0.06008225, -0.05984951, -0.05962376,\n",
       "        -0.05961432, -0.05946671, -0.05997718, -0.05970079, -0.05951859,\n",
       "        -0.05940331, -0.05965666, -0.0595626 , -0.05979836, -0.0596377 ,\n",
       "        -0.0594643 , -0.05960582, -0.05929285, -0.0596385 ]),\n",
       " 'std_fit_time': array([0.11436893, 0.16324703, 0.13306854, 0.18873202, 0.29410869,\n",
       "        0.3660208 , 0.44646764, 0.18224879, 0.2701889 , 0.46355343,\n",
       "        0.35637572, 0.19314424, 0.39091164, 0.43875203, 0.16382073,\n",
       "        0.1289836 , 0.2275035 , 0.10302225, 0.19319388, 0.27880553,\n",
       "        0.28682631, 0.28599616, 1.70343322, 0.29577576]),\n",
       " 'std_score_time': array([0.00278237, 0.02655843, 0.00168688, 0.00203064, 0.00400074,\n",
       "        0.00394305, 0.00989081, 0.00147187, 0.00663411, 0.00867162,\n",
       "        0.00808368, 0.00300531, 0.00682704, 0.00253395, 0.0017515 ,\n",
       "        0.00128036, 0.00711183, 0.00112478, 0.00269805, 0.00176417,\n",
       "        0.00179423, 0.0093764 , 0.00475225, 0.00139486]),\n",
       " 'std_test_score': array([0.0011053 , 0.00104852, 0.00120729, 0.00108008, 0.00114428,\n",
       "        0.00121282, 0.0011159 , 0.00101548, 0.00111613, 0.0010078 ,\n",
       "        0.00102847, 0.00103407, 0.00109408, 0.00118649, 0.00096736,\n",
       "        0.0012331 , 0.00121838, 0.00116414, 0.00115574, 0.00119115,\n",
       "        0.00105551, 0.00118468, 0.00100572, 0.00102269]),\n",
       " 'std_train_score': array([0.00030361, 0.00029396, 0.00015912, 0.00040775, 0.00025182,\n",
       "        0.00027385, 0.00021968, 0.0004439 , 0.00029147, 0.00035881,\n",
       "        0.00026699, 0.00040594, 0.00021451, 0.00030085, 0.00022551,\n",
       "        0.00025838, 0.00037005, 0.00031415, 0.00035279, 0.00027755,\n",
       "        0.00034683, 0.00030046, 0.00037398, 0.00037977])}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch3_1.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.063638 using {'subsample': 0.8, 'colsample_bytree': 0.8}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzsnXlYVdX6xz+LQRBFUCZxxDlHEAdynlKz1HLIUisry8zUhltp9fNa3uraaA5ZOeW9aTlmWnadck4DJ0TFCdEUB0QQVJTxvL8/9pEQQY7I4RxgfZ6H55y991prv3sD+7vXWu96XyUiaDQajUZT2DjY2gCNRqPRlEy0wGg0Go3GKmiB0Wg0Go1V0AKj0Wg0GqugBUaj0Wg0VkELjEaj0WisghYYjUaj0VgFLTAajUajsQpaYDQajUZjFZxsbYAt8fb2loCAAFubodFoNMWKPXv2XBIRn/zKlWqBCQgIYPfu3bY2Q6PRaIoVSqm/LCmnh8g0Go1GYxW0wGg0Go3GKmiB0Wg0Go1VKNVzMBqNvZCenk5MTAwpKSm2NkWjycLV1ZVq1arh7OxcoPpaYDQaOyAmJgZ3d3cCAgJQStnaHI0GESE+Pp6YmBhq1apVoDb0EJlGYwekpKTg5eWlxUVjNyil8PLyuqdetRYYjcZO0OKisTfu9W9SC0wB2Hv6Ml9vPmFrMzQajcau0QJTAFaFn+PjNUdYe+iCrU3RaDQau0ULTAF4+6H7aFbNgzeW7ud0/HVbm6PR3DOJiYnMnDmzQHW//PJLrl8vvP+D+fPnM3r06EJrz1I2b95M796977re3dp76tQpfvjhh7s+j6UkJCTQvXt36tWrR/fu3bl8+XKu5U6fPk2PHj1o2LAhjRo14tSpU4VuixaYAuDi5MhXQ4JRwEsL95CSnmlrkzSae8KeBKakcyeBycjIuOf2J0+eTLdu3Th+/DjdunVj8uTJuZZ7+umnefPNNzl8+DBhYWH4+vre87lzot2UC0j1Sm58PiiIF/67m0m/RvJRv6a2NklTQnj/l0NEnrtSqG02qlKBiX0a53l8/PjxnDhxgqCgILp3746vry9LliwhNTWVfv368f7775OcnMygQYOIiYkhMzOTCRMmEBsby7lz5+jSpQve3t5s2rQp1/bXrFnDO++8Q2ZmJt7e3vz+++8kJCTw3HPPER0djZubG7NmzaJZs2a31Fu6dCnvv/8+jo6OeHh4sHXrVk6dOsVTTz1FcnIyADNmzKBt27Zs3ryZiRMn4ufnR3h4OP3796dp06ZMnTqVGzdu8PPPP1OnTh2eeeYZXF1dOXToELGxsXzxxRe39VySk5MZM2YMBw4cICMjg/fee49HHnkkz/t35swZHnzwQU6ePMmQIUOYOHEiEyZMwNvbm1deeQWAd999Fz8/P3744QcOHz5MUFAQw4YNo2LFiqxevZqUlBSSk5PZuHEjn3766W33H2DBggVMmzaNtLQ0QkJCmDlzJo6OjrfYsnLlSjZv3gzAsGHD6Ny5Mx9//PEtZSIjI8nIyKB79+4AlC9fPs9ruxe0wNwD3Rv58WLH2ny7NZrWAZV4tHlVW5uk0RSIyZMnc/DgQcLDw1m3bh3Lli0jLCwMEaFv375s3bqVuLg4qlSpwurVqwFISkrCw8ODL774gk2bNuHt7Z1r23Fxcbzwwgts3bqVWrVqkZCQAMDEiRNp3rw5P//8Mxs3buTpp58mPDz8lrqTJk1i7dq1VK1alcTERAB8fX1Zv349rq6uHD9+nMGDB2cFrd2/fz+HDx+mUqVK1K5dm+eff56wsDCmTp3K9OnT+fLLLwGjF7FlyxZOnDhBly5diIqKuuW8H374IV27dmXevHkkJibSunVrHnjgAcqVK5frNYaFhXHw4EHc3Nxo1aoVDz/8MMOHD6d///688sormEwmFi1aRFhYGM2aNeOzzz7j119/BYwhtp07dxIREUGlSpVYt24dx48fv+3++/j4sHjxYv744w+cnZ0ZNWoUCxcu5Omnn+b5559n5MiRtGzZktjYWPz9/QHw9/fn4sWLt9l77NgxPD096d+/PydPnuSBBx5g8uTJt4nVvaIF5h55o2cD9p6+zNs/HaBxlQrU83O3tUmaYs6dehpFwbp161i3bh3NmzcH4Nq1axw/fpwOHTrwxhtvMG7cOHr37k2HDh0sau/PP/+kY8eOWYv1KlWqBMD27dtZvnw5AF27diU+Pp6kpKRb6rZr145nnnmGQYMG0b9/f8CIejB69GjCw8NxdHTk2LFjWeVbtWqV9XCtU6cOPXr0AKBp06a39K4GDRqEg4MD9erVo3bt2hw5cuS2e7Bq1So+++wzwFindPr0aRo2bJjrNXbv3h0vLy8A+vfvz/bt23n11Vfx8vJi3759xMbG0rx586wyudW/eV/yuv8RERHs2bOHVq1aAXDjxo2sYa05c+bk2m5eZGRksG3bNvbt20eNGjV4/PHHmT9/PsOHD7+rdvJDC8w94uzowPTBwTw8bRujFu5l5eh2uJXRt1VTfBER3n77bV588cXbju3Zs4fffvuNt99+mx49evDPf/7TovZyW08hIrfty1num2++ITQ0lNWrVxMUFER4eDjTp0/Hz8+P/fv3YzKZcHV1zSrv4uKS9d3BwSFr28HB4Zb5jZznybktIixfvpwGDRrke313au/5559n/vz5XLhwgeeeey7P+tl7Rnnd/+nTpzNs2DD+/e9/39EWPz8/zp8/j7+/P+fPn891bqVatWo0b96c2rVrA/Doo4/y559/FrrA6En+QqCyhytTn2hOVNw13l1xMNd/HI3GnnF3d+fq1asA9OzZk3nz5nHt2jUAzp49y8WLFzl37hxubm48+eSTvPHGG+zdu/e2urnRpk0btmzZwsmTJwGyhsg6duzIwoULAcODy9vbmwoVKtxS98SJE4SEhDBp0iS8vb05c+YMSUlJ+Pv74+DgwPfff09m5t072SxduhSTycSJEyeIjo6+TUh69uzJ9OnTs/6X9+3bd8f21q9fT0JCQtZcT7t27QDo168fa9asYdeuXfTs2RPI/37ldf+7devGsmXLsoa8EhIS+Ouv29Oy9O3bl//85z8A/Oc//8l17qhVq1ZcvnyZuLg4ADZu3EijRo3ueI0FQb9qFxLt63nzarf6TNlwjFYBlRgSUsPWJmk0FuPl5UW7du1o0qQJvXr1YsiQIbRp0wYwJoAXLFhAVFQUb775Jg4ODjg7O/P1118DMGLECHr16oW/v3+uk/w+Pj7MmjWL/v37YzKZsuZQ3nvvPZ599lmaNWuGm5tb1kMxO2+++SbHjx9HROjWrRuBgYGMGjWKAQMGsHTpUrp06ZLnvMidaNCgAZ06dSI2NpZvvvnmll4QwIQJE3j11Vdp1qwZIkJAQEDWnElutG/fnqeeeoqoqCiGDBlCy5YtAShTpgxdunTB09Mza36jWbNmODk5ERgYyDPPPEPFihVvaatHjx4cPnz4tvvfqFEjPvjgA3r06IHJZMLZ2ZmvvvqKmjVr3jIHM378eAYNGsTcuXOpUaMGS5cuBWD37t188803zJkzB0dHRz777DO6deuGiNCiRQteeOGFu76P+aFK89t2y5YtpTAzWppMwrDvwgg9mcBPL7WlSVWPQmtbU7I5fPhwnuP7msLlmWeeoXfv3gwcONDq5zKZTAQHB7N06VLq1atn9fNZg9z+NpVSe0SkZX51rTpEppR6UCl1VCkVpZQan8txF6XUYvPxUKVUQLZjzZRSO5VSh5RSB5RSrjnqrlJKHcylzTeUUqKUyt2lxYo4OCi+fDyISm5lGLVwL0k30ovaBI1GYydERkZSt25dunXrVmzF5V6x2hCZUsoR+AroDsQAu5RSq0QkMlux4cBlEamrlHoC+Bh4XCnlBCwAnhKR/UopLyA9W9v9gWu5nLO6+XynrXVd+eFV3oWvhjbn8W//5M2l+/n2qRY6iKGm1BASEkJqauot+77//nuaNrWfdWLz588vUL21a9cybty4W/bVqlWLFStW5Fq+UaNGREdHF+hcJQVrzsG0BqJEJBpAKbUIeATILjCPAO+Zvy8DZijjadwDiBCR/QAiEn+zglKqPPA6MAJYkuOcU4C3gJWFfTF3Q4ualRjf6z4+WH2YudtP8nyH2rY0R6MpMkJDQ21tgtXo2bNn1kS9xjKsOURWFTiTbTvGvC/XMiKSASQBXkB9QJRSa5VSe5VSb2Wr8y/gc+CW2BRKqb7A2ZuilBdKqRFKqd1Kqd03PSiswfD2tejRyI/J/zvCnr8SrHYejUajsVesKTC5jQvl9CjIq4wT0B4Yav7sp5TqppQKAuqKyC19UqWUG/AukK9TvojMEpGWItLSx8fHgssoGEopPn0skCqeZXl54T7ir6XmX0mj0WhKENYUmBigerbtasC5vMqY5108gATz/i0icklErgO/AcFAG6CFUuoUsB2or5TaDNQBagH7zceqAXuVUpWtcmUW4lHWmZlDg0m4nsZrS/ZjMpVejz2NRlP6sKbA7ALqKaVqKaXKAE8Aq3KUWQUMM38fCGwUw296LdBMKeVmFp5OQKSIfC0iVUQkAKNnc0xEOovIARHxFZEA87EYIFhEbJ6wpUlVDyb2acTWY3HM2BSVfwWNRqMpIVhNYMxzKqMxxOIwsEREDimlJpnnSwDmAl5KqSiMifvx5rqXgS8wRCoc2Csiq61lq7UZ0roGjwZVYcqGY/wRdcnW5mg0t2FP4fp1Pph7w5J8MJs2bSIoKCjrx9XVlZ9//rnQbbHqOhgR+U1E6otIHRH50LzvnyKyyvw9RUQeE5G6ItL6pseZ+dgCEWksIk1E5K1c2j4lIk3yOG+AiNjNk1wpxYf9mlLHpzyvLNpH7JUUW5uk0dyCPQlMScce8sF06dKF8PBwwsPD2bhxI25ublmBQQsTHSqmiCjn4sTXQ4PpO+MPxvywjx9eCMHJUYeC0+TC/8bDhQOF22blptAr98RToPPBlLZ8MNlZtmwZvXr1ws3NLc8yBUULTBFSz8+df/dvyquLw/l03VHe7qVDg2jsA50PpnTlg8nOokWLeP311+9YpqBogSliHm1elbBTCXy7JZqWNSvRvZGfrU3S2Bt36GkUBTofTMnPB3OT8+fPc+DAAastINUCYwP+2bsRETGJ/GNJOKvHdqB6pcLvmmo0BUXngyn5+WBusmTJEvr164ezs/Md2ywoehLABrg6OzJzSAsEePmHvaRm3H0+C42mMNH5YEpXPpib/PjjjwwePPiO13Yv6B6Mjajh5canAwMZuWAPH64+zKRHcnWI02iKBJ0PpnTlgwFjHurMmTN06tTpru+fpeh8MIWYD6YgfPBrJHO2n2T64Ob0CaxiU1s0tkPngyk6dD6Yu8Nu88Fo8mdcr/toUbMi45dHcCLutgwEGo2mmKLzwegejM17MADnk27w8LTt+JR34eeX21G2jGP+lTQlipLSgykO+WAKyt3mgykp3EsPRs/B2AH+HmX58vEghn0Xxv/9fJDPHmumk5RpiiU6H4wmO3qIzE7oWN+HMV3rsXxvDEt2n8m/gkaj0dg5WmDsiFe61aN9XW/+ufIQkeeu2NocjUajuSe0wNgRjg6KL58IwtPNmVEL93A1Jd3WJmk0Gk2B0QJjZ3iXd2HGkGDOXL7BuOURua521mgKG3uKpqzD9d8bloTrB3jrrbdo3LgxDRs2ZOzYsVZ51miBsUNaBVTirZ4N+O3ABebvOGVrczSlAHsSmJKOPYTr37FjB3/88QcREREcPHiQXbt2sWXLlns+d06sKjBKqQeVUkeVUlFKqfG5HHdRSi02Hw9VSgVkO9ZMKbVTKXVIKXVAKeWao+4qpdTBbNv/UkpFKKXClVLrlFLFetXiiI61eaChLx/9dph9p3N/A9FoCovs4frffPNNPv30U1q1akWzZs2YOHEiYISwf/jhhwkMDKRJkyYsXryYadOmZYXr79KlS57tr1mzhuDgYAIDA+nWrRtgvGk/+uijNGvWjPvvv5+IiIjb6i1dupQmTZoQGBhIx44dAeMB3aFDB4KDgwkODmbHjh2A0QPp1KkTgwYNon79+owfP56FCxfSunVrmjZtyokTJwBjoeXIkSPp0KED9evXz3WFfnJyMs899xytWrWiefPmrFy58o7372a4/gYNGmSF1p8wYQJTp07NKvPuu+8ybdo0xo8fz7Zt2wgKCmLKlCnMnz+fxx57jD59+mQF58zt/oMRrr9169YEBQXx4osv5homZ+XKlQwbZiQKHjZsWK6JxJRSpKSkkJaWRmpqKunp6fj5FX7gXau5KSulHIGvgO4YKYx3KaVWiUhktmLDgcsiUlcp9QTwMfC4OU3yAuApEdmvlPIC0rO13R/IuSrxUxGZYD4+FvgnMNJKl2d1lFJ8/lgQD0/fxssL97J6bAcqlitja7M0RcDHYR9zJOFI/gXvgvsq3ce41uPyPK7D9ZeucP1t2rShS5cu+Pv7IyKMHj3aKuuwrNmDaQ1EiUi0iKQBi4CcUdceAW4GIFoGdFPGApAeQISI7AcQkXgRyQRQSpXHSK/8QfaGRCS721U5oNhPXni4OTNzaDCXrqXx2pJwTKZif0maYkD2cPHBwcEcOXKE48eP07RpUzZs2MC4cePYtm0bHh4eFrV3p3D9Tz31FJB/uP7Zs2dnva2np6fzwgsv0LRpUx577DEiI/9+Z70Zrt/FxeW2cP2nTp3KKmdJuP7JkycTFBRE586ds8L158XNcP1ly5bNCtcfEBCQFa7/5v2823D92e//77//nhWuPygoiN9//53oaCMJ8Jw5c7Lin1lCVFQUhw8fJiYmhrNnz7Jx40a2bt1qcX1LseZCy6pA9gUdMUBIXmVEJEMplQR4AfUBUUqtBXyARSLyibnOv4DPgdsGfZVSHwJPA0lA3v31YkSzap5M6N2QCSsP8fWWE7zcpa6tTdJYmTv1NIoCHa6/5IfrX7FiBffffz/ly5cHoFevXlkvAoWJNXswuS1Fz/kXlVcZJ6A9MNT82U8p1U0pFQTUFZFcYzOIyLsiUh1YCOTq1qGUGqGU2q2U2h0XF2fhpdiWJ++vSZ/AKny+7ig7TlyytTmaEogO11+6wvXXqFGDLVu2kJGRQXp6Olu2bLHKEJk1ezAxQPVs29WAc3mUiTHPu3gACeb9W0TkEoBS6jcgGGPepYVS6pTZdl+l1GYR6Zyj3R+A1cDEHPsRkVnALDBikd3D9RUZSin+3b8ph84lMfbHcH4b2x7fCq75V9RoLESH6y9d4foHDhzIxo0badq0KUopHnzwQfr06XPX9zFfRMQqPxgCEA3UAsoA+4HGOcq8DHxj/v4EsMT8vSKwF3Azt7MBeDhH3QDgYLbtetm+jwGW5WdjixYtpDhx5PwVafB/v8mgb3ZIekamrc3RFCKRkZG2NqHUMGzYMFm6dGmRnCszM1MCAwPl2LFjRXI+a5Db3yawWyzQAasNkYlIBsYw1VrgsFk8DimlJiml+pqLzQW8lFJRGBP34811LwNfALuAcGCviKzO55STlVIHlVIRGE4CrxT6RdmYBpXd+fDRpoSeTGDKhmP5V9BoNDZDh+vX4frtIlz/3TJ+eQSLdp3hu2da0eW+vPNta4oPOly//aPD9f+NDtdfgnmvb2MiYpJ4bUk4q8d2oKpnWVubpNEAOly/5lZ0qJhiiKuzIzOHBpOZKby8cC9pGSZbm6TRaDS3oQWmmBLgXY5PBjYj/EwiH/122NbmaDQazW1ogSnG9Grqz7PtApi/4xSrI87b2hyNRqO5BS0wxZy3ezWkeQ1Pxi2PIDouZ3g2jUajsR1aYIo5ZZwcmDEkGCdHxaiFe0lJv/tVzRqNPYXr1/lg7g1L88GMGzeOJk2aZEXGtgZaYEoAVT3LMuXxII5cuMrElYdsbY6mGGJPAlPSsYd8MKtXr2bv3r2Eh4cTGhrKp59+ypUrhZ+mXQtMCaFLA19Gd6nL4t1nWLYnxtbmaIoZOh/MrZT0fDCRkZF06tQJJycnypUrR2BgIGvWrLnjNRYEvQ6mBPFa9/rs+esy//fzAZpUrcB9lSvkX0ljd1z46CNSDxduPhiXhvdR+Z138jyu88GUrnwwgYGBvP/++7z++utcv36dTZs20ahRozz/PgqKFpgShKODYurgIB6etp1RC/eyanR7yrvoX7Hm7siejwTg2rVrHD9+nA4dOvDGG28wbtw4evfuTYcOHSxq7075YJYvXw7knw9m0KBB9O/fHzDywYwePZrw8HAcHR05duzvsEk388EAt+WDyR6I05J8MKtWreKzzz4DyMoHk1e0hZv5YICsfDCvvvpqVj6Y2NjYAuWDgb/vf0RERFY+GIAbN25kheKfM2dOru3mRY8ePdi1axdt27bFx8eHNm3a4ORU+M8K/fQpYfi6uzJ9cHOGzP6T8csjmD64ea65ODT2y516GkWB6HwwJT4fDBhDdu+++y4AQ4YMsUq8ND0HUwK5v7YXb/RswK8R5/n+z9vzRWg0OdH5YEpXPpjMzEzi4+MBiIiIICIiIqu3V5joHkwJZWTHOuw5dZl//RpJYDVPAqt72tokjR2j88GUrnww6enpWUOcFSpUYMGCBVYZItPRlIthNGVLSbyexsPTtgOwemx7PN3K2NgiTV6UlGjKxYFnnnmG3r17M3DgQKufy2QyERwczNKlS4ttyP57iaash8hKMJ5uZfhqaDAXr6bwjyX7MZlK78uERlPU6HwwVh4iU0o9CEwFHIE5IjI5x3EX4L9ACyAeeFxETpmPNQO+BSoAJqCViKRkq7sKqC0iTczbnwJ9gDTgBPCsiCRa8/qKA0HVPXn3oYa890sks7ZFM7JTHVubpCnBFId8MPPnzy9QvbvNB9OoUSOio6MLdK6SgtUERinlCHwFdAdigF1KqVUiEpmt2HDgsojUVUo9AXwMPK6UcgIWAE+JyH6llBeQnq3t/kDOwFvrgbdFJEMp9THwNjAODcPaBrDr1GU+XXuU5tU9Camdu6ukRnOv6HwwmuxYc4isNRAlItEikgYsAnK6MzwC3JzZWwZ0U4Z/Xw8gQkT2A4hIvIhkAiilymOkV/4ge0Miss6cphngT6CaFa6pWKKUYvKAptSo5MaYH/cRdzU1/0qaIqc0z4dq7JN7/Zu0psBUBc5k244x78u1jFkckgAvoD4gSqm1Sqm9Sqm3stX5F/A5cKfgR88B/7s380sW7q7OzBwaTNKNdF5ZtI9MPR9jV7i6uhIfH69FRmM3iAjx8fG3edjdDdacg8ltdV/O/568yjgB7YFWGELyu1JqD8Y8TV0ReU0pFZDrSZV6F8gAFuZxfAQwAqBGjRr5XkRJoqF/Bf71aBPeWhbB1A3HeL2HZYvINNanWrVqxMTEEBcXZ2tTNJosXF1dqVat4INB1hSYGKB6tu1qwLk8ysSY5108gATz/i0icglAKfUbEIwx79JCKXXKbLuvUmqziHQ2lxsG9Aa6SR6vgiIyC5gFhpvyvV9m8WJQy+rsOpnA9E1RtAioRKf6PrY2SQM4OztnhVLRaEoK1hwi2wXUU0rVUkqVAZ4AVuUoswoYZv4+ENhoFoa1QDOllJtZeDoBkSLytYhUEZEAjB7OsWzi8iDGpH5fEdGxw+/ApEea0MDPnVcX7eNc4g1bm6PRaEooVhMY85zKaAyxOAwsEZFDSqlJSqm+5mJzAS+lVBTGxP14c93LwBcYIhUO7BWR1fmccgbgDqxXSoUrpb4p9IsqIZQt48jMocGkZwqjf9hLeqbJ1iZpNJoSiF7JX4JX8ufH6ojzvPzDXoa3r8WE3oUfqluj0ZRM9Ep+Tb483MyfZ9oGMHf7SdYcvGBrczQaTQlDC0wp5+2H7iOwmgdvLt3PX/HJtjZHo9GUILTAlHJcnBz5amgwDg6KUQv3kpJ+96HPNRqNJje0wGioVtGNKY8HcujcFd7/JTL/ChqNRmMBWmA0AHS9z4+XOtfhx7DTrNgXY2tzNBpNCUALjCaLf3SvT0itSrzz00GOxeadcU+j0WgsQQuMJgsnRwemD25OORcnRi3cS3JqRv6VNBqNJg+0wGhuwbeCK9MGBxEdd413VhzQwRc1Gk2B0QKjuY22dbx5vXt9VoafY2HoaVubo9FoiilaYDS5MqpzXTo38GHSL5EcPJtka3M0Gk0xRAuMJlccHBRTBgXhXb4MLy3cQ9KN9PwraTQaTTa0wGjypGK5MswYGsz5xBTeXLpfz8doNJq7Il+BUUrVUUq5mL93VkqNVUp5Wt80jT0QXKMibz/UkHWRsXy+7piOvKzRaCzGkh7MciBTKVUXI7x+LeAHq1qlsSueaxfAo0FVmLEpip5fbmXtoQu6N6PRaPLFEoExmXO79AO+FJHXAH/rmqWxJ5RSTHk8iNlPt8RBKV78fg8Dv9nJnr8SbG2aRqOxYywRmHSl1GCMzJO/mvc5W9K4UupBpdRRpVSUUmp8LsddlFKLzcdDlVIB2Y41U0rtVEodUkodUEq55qi7Sil1MNv2Y+ayJqVUvnkKNHeHUorujfxY80oH/t2/KacTrjPg652M/H4P0XHXbG2eRqOxQywRmGeBNsCHInJSKVULWJBfJaWUI/AV0AtoBAxWSuXMajUcuCwidYEpwMfmuk7mc4wUkcZAZyDLjUkp1R/I+VQ7CPQHtlpwTZoC4uTowODWNdjyZmde716fbcfj6D5lK//38wHirqba2jyNRmNH5CswIhIpImNF5EelVEXAXUQmW9B2ayBKRKJFJA1YBDySo8wjwH/M35cB3ZRSCugBRIjIfrMN8SKSCaCUKo+RXvmDHHYeFpGjFtilKQTcyjgxtls9trzVhaEhNVgUdobOn25i6objOsSMRqMBLPMi26yUqqCUqgTsB75TSn1hQdtVgTPZtmPM+3ItY57nSQK8gPqAKKXWKqX2KqXeylbnX8DnwHULbNBYGe/yLkx6pAnrXutIx/o+TNlwjM6fbWZh6F9kaI8zjaZUY8kQmYeIXMEYfvpORFoAD1hQT+WyL6frUV5lnID2wFDzZz+lVDelVBBQV0RWWHD+3I1SaoRSardSandcXFxBm9HkoLZPeb5+sgXLX2pLgJcb7644SM8vt7JOe5xpNKUWSwQmNWupAAAgAElEQVTGSSnlDwzi70l+S4gBqmfbrgacy6uMed7FA0gw798iIpdE5DrwGxCMMRfUQil1CtgO1FdKbb4LmxCRWSLSUkRa+vj43E3VLK6nXyc6MbpAdUs6LWpWZMmLbZj1VAsEGPH9HgZ9u5O9py/b2jSNRlPEWCIwk4C1wAkR2aWUqg0ct6DeLqCeUqqWUqoM8ASwKkeZVRjeaQADgY1ivO6uBZoppdzMwtMJiBSRr0WkiogEYPRsjolIZwtsKVQ+DP2Qp9c8TWS8zv6YG0opejSuzLpXO/JRv6acir9O/5k7eGmB9jjTaEoTyprDF0qph4AvAUdgnoh8qJSaBOwWkVVm1+PvgeYYPZcnRCTaXPdJ4G2MIbPfROStHG0HAL+KSBPzdj9gOuADJALhItLzTva1bNlSdu/efdfXFXM1huFrh3M1/Sqzu8+msXfju26jNJGcmsGcbSeZtfUEqRkmBreuwdhu9fBxd7G1aRqNpgAopfaISL7LQfIVGKVUNYwHdzuMh/124BURKfZ5dQsqMABnr51l+NrhXEm9wjfdv6GZT7NCtq7kEXc1lWm/H+eHsNO4OjnwYqc6PN+hFm5lnGxtmkajuQssFRhLhsi+wxjKqoLh9fWLeV+ppmr5qnzX8zs8XT0ZsX4E4RfDbW2S3ePj7sK/HjU8zjrU8+GL9cfo9Olmfgg9rT3ONJoSiCUC4yMi34lIhvlnPsYwVKnHv7w/83rOw7usNy+uf5G9sXttbVKxoI5Peb55qgXLX2pDjUpuvLPiAD2/3Mr6yFjtcabRlCAsEZhLSqknlVKO5p8ngXhrG1ZcqFyuMvN6zsPXzZeRG0ay68IuW5tUbGhRsxLLRrbh26daIAIv/Hc3j3/7J/u0x5lGUyKwRGCew3BRvgCcx/D2etaaRhU3fN18+e7B7/Av58+oDaMIPR9qa5OKDUopejauzNrXOvLBo02IvpRMv5k7GLVwDycvJdvaPI1Gcw8UyItMKfWqiHxpBXuKlHuZ5M+N+BvxPL/uec5cPcO0rtNoW6VtobVdWkhOzWD2tmhmbY0mLcPE0JAajOlWD+/y2uNMo7EXCs2LLI/GT4tIjQJZZkcUWGCOrYOLkeBZHTxqGJ/lfMHBgYSUBF5Y9wKnkk4xtetU2ldtX/iGlwIuXk1h6objLNp1BlcnB0Z2qsNw7XGm0dgF1haYMyJSPf+S9k2BBebX12H33Fv3ObqAR1XwqM7lCv6MSDnMifQrfNl4JB1r94IKVcFRPxzvlhNx1/hkzRHWHorF192F17rX57EW1XBy1Nm+NRpboXswFnBPQ2QpVyApBhJPQ9KZbJ9nIOkMSdfjeKGyL8fLODMl9hKdU1LBvYq511MdPGvc+t2jGjiXLdwLLEHsPpXAR78dZu/pROr6lmf8g/fRraEvRvBtjUZTlNyzwCilrnJ7cEowAlSWFZFi/zpe2HMwt5CeQtKlo4zc8Q5HkmP4rFIbuqXJ3yJ05SwYGQj+ppyPWXBuCk/NbN+rg6uHdWwtJogIaw9d4JM1R4m+lEzrWpV4u9d9NK9R0damaTSlCqv2YEoKVhUYM1fTrjJyw0giL0XySadP6F6zu3EgMwOuns/W6zltfN7sCSXFQEbKrY25eNwqOLf0hGpAOW8oBW/06ZkmFu06w9QNx7h0LY2Hm/rzZs8GBHiXs7VpGk2pQAuMBRSFwABcS7vGSxte4sClA0zuOJkHAx7Mv5IIJMfdKj7ZhuBIPA2pV26t41TWGGq7RYRq/C1C7v7g4Gidi7QB11IzmL3V8DhLzzTx5P01GdO1Ll7a40yjsSpaYCygqAQGIDk9mVEbRhEeF85H7T/i4doP33ujNxJvF53s28k58t04OEGFKn97vuXsCXlUA6fi93C+eCWFL38/zuJdZyjr7MhLnevwXLtalC1TcsRUo7EntMBYQFEKDBh5ZEZvHM2e2D180O4D+tTpY90Tpl03htpy7QGdgavnQHLEACtfOQ/xqQ5edexagKIuXuXjNUdZHxmLXwUXXu9en4EtquPoUPKHDTWaokQLjAUUtcAA3Mi4wZiNYwg7H8akdpN4tO6jRXr+W8hMhyvn/u795ByOS4qBzLS/y5fzhc7jIfhpcHS2nd35sMvscbbvdCL1/cozvtd9dGmgPc40msKiMMP15+ZNlgTsBv5xM39LccQWAgOQkpHCK5teYee5nUxsM5EB9QcUuQ0WYTJB8kWz88FfsHse/PUHeNWFbhOhYR+7dSoQEdYcvMDHa45wKv46IbUq8c5DDQms7mlr0zSaYk9hCsz7GKmOf8BwUX4CqAwcBV6yRUbJwsJWAgOQmpnKK5te4Y+zfzDh/gkMajDIJnbcFSJwbA1seA/ijkC11tB9EtRsY2vL8iQ908SPYaeZuuE48clp9G5meJzV9NIeZxpNQSlMgQkVkZAc+/4UkfuVUvtFJPAOdR8EpmJktJwjIpNzHHcB/gu0wIjQ/LiInDIfawZ8C1QATEArEUnJVncVUDtbRstKwGIgADgFDBKRO4bltaXAAKRlpvHa5tfYGrOVd0LeYfB9g21my12RmQH7f4BNHxmu1g0ehgcmgk8DW1uWJ1dT0pm9NZrZ206SYTIxNKQmY7vVo1K5MrY2TaMpdhRmwjGTUmqQUsrB/JP9VTtPdVJKOQJfAb2ARsBgpVSjHMWGA5dFpC4wBfjYXNcJWACMFJHGQGcgPVvb/YGcyd3HA7+LSD3gd/O2XVPGsQxTOk+hS/UufBT6EQsiF9jaJMtwdDLmYcbsha4T4NQ2mHk/rBoLV87b2rpccXd15vUeDdj8ZmcGtqjGf3eeotMnm/hqUxQ30jLzra/RaO4eS3owtTF6ITfHQXYCrwFngRYisj2Pem2A90Skp3n7bQAR+Xe2MmvNZXaaReUCRjKzXsAQEXkyl3bLA2uAEcCSbD2Yo0BnETmvlPIHNovIHV+pbd2DuUl6ZjpvbX2LDac38EbLNxjWeJitTbo7kuNh66ewa44x+d/mZWg7Flwr2NqyPDkea3icbTgcS+UKrrzevT4DWlTTHmcajQUUWg9GRKJFpI+IeJt/+ohIlIjcyEtczFQFzmTbjjHvy7WMiGRgOA94AfUBUUqtVUrtVUq9la3Ov4DPges52vITkfPmts4DvrkZpZQaoZTarZTaHRcXl1uRIsfZ0ZlPOn1Cj5o9+Gz3Z8w7OM/WJt0d5byg12QYvQsaPGSIzbQgCP0WMtLyr28D6vm5M2dYSxaPuB8/D1feWh7BQ1O3senIRZ1VU6MpJPIVGKVUNaXUCqXURaVUrFJquVKqmgVt5/YqmPM/N68yTkB7YKj5s59SqptSKgioKyIrLDh/rojILBFpKSItfXzsJ/Ozs4MzH3f8mF4BvZiyZwqzI2bb2qS7p1ItGDgXXtgEfo3hf2/BV63h4E+Gg4AdElLbi59HtWXm0GBSMzJ5dv4uBs/+k4iYRFubptEUeyyZg/kOWAVUwehx/GLelx8xQPaQ/tUwvNFyLWMeIvMAEsz7t4jIJRG5DvwGBGMM07VQSp0CtgP1lVKbzW3FmofGMH9etMBGu8LJwYmPOhir/Kftm8bX+7+2tUkFo2owPL0Khi6HMuVg2bMwuyuc3GZry3JFKcVDTf1Z91on3u/bmOOx1+g74w/G/LiPUzqrpkZTYCwRGB8R+U5EMsw/8zHmSfJjF1BPKVVLKVUGw715VY4yq4CbEw4DgY1ijE+sBZoppdzMwtMJiBSRr0WkiogEYPRsjmVzk87e1jBgpQU22h1ODk582O5D+tbpy8zwmczYN6N4DtkoBfUegBe3wqNfw7WL8J/esPAxiD1ka+typYyTA8PaBrD5zc6M6VqX9ZEX6Pr5Zl5bHE7Uxau2Nk+jKXZYEnL/klLqSeBH8/ZgDJfiOyIiGUqp0Rhi4QjME5FDSqlJwG4RWQXMBb5XSkVh9FyeMNe9rJT6AkOkBPhNRFbnc8rJwBKl1HDgNPCYBddmlzg6ODKp7SQclSPfRnyLSUyMaT6meK5Ed3CEoCHQuB+EzYJtn8PX7Yx9Xd4x4p/ZGe6uzvyjRwOeur8ms7dFs+DP0/wcfpaHmvjzcpe6NKpiv84LGo09YYkXWQ1gBsbwlAA7gLEictr65lkXe/EiywuTmJi0cxLLjy/n2SbP8lrwa8VTZLJzPcEQmbBZoBwgZCS0fw3K2u8K+/hrqcz74yT/2fEX11IzeKChH6O71iVIRwXQlFKsndHyVRH5skCW2RH2LjBgiMxHoR+x+Ohinm70NG+0fKP4iwwYsc82fggRiw1x6fgmtHreroNpJl1PZ/6OU8z74yRJN9LpUM+bsd3q0Sqgkq1N02iKFJ0y2QKKg8CAEVdrcthkfjjyA082fJK3Wr1VMkQG4HwEbJgIJzYaUZu7ToAmA8HBkulB23AtNYPvd/7FnG3RxCenEVKrEmO61qNdXa+S83vRaO6AtQXmjIhUz7+kfVNQgZG0NHB2LtKHiYjwya5PWHB4AU80eIJ3Qt4pWQ+zE5tg/T/hQgRUbgbd34c6XW1t1R25kZbJj2Gn+XbrCWKvpNK8hidjutbVkZs1JR7dg7GAggrMhY8+InHxEpwq++HsVxknPz+cK/vh5FfZ/Gn+8fJCORZe0isR4Ys9XzD/0HwG1R/Eu/e/i4Oy3zf9u8ZkgoPLYeMkYwitTld44H3wb2Zry+5ISnomy/bE8PXmE5xNvEHjKhUY07UuPRpVxkFHBtCUQO5ZYPII0w/G4siyImKJB5pdU1CBubp5M9dDw8iIvUB67EUyLlwg/eJFSE+/taCTE06+Pn+LkJ8fTpX/FiFnPz+cfHxQZSwPuCgiTN07lbkH5zKg3gD+2eafJUtkADJSjbAzWz81snY2GwRd/88YQrNj0jNNrNh3lpmbojgVf516vuUZ3bUuvZtV0SFoNCUKnXDMAgpzDkZMJjITEkiPjSUjNpb0CxfIuBB7qwjFxiI3btxaUSkcvb1w9jWLT3YR8vXLEiOHsmX/PpcIM8JnMCtiFo/UeYT3276Po0MJTA98IxH++BL+/NrIvNl6BHT4B7jZ96R6RqaJ1QfOM2NjFMcvXqOWdzle6lyHfs2r4uxYwl4GNKUSLTAWUNST/CKC6cqV20XoYizpF2KzRMh05cptdR09PIxht5vDcpX92J56mOVJm2jasAuv9foA5woeJXPsP+mskRogfKERQLP96xDyIjiXzb+uDTGZhHWRF5i+MYpD565Q1bMsL3Wuw2Mtq+HiVAJfCDSlBi0wFmCvXmSm69dvFaHYi0ZPKJsIZcbfvtZVubmZe0B/i5CzX7a5ocqVcaxYsfiKUGykkezs+FqoUM1YqBn4hLGY044RETYdvci036MIP5OIXwUXXuxYh8Gta1C2jH3brtHkhhYYC7BXgbEEU1oaGRfjyIi9wLpdPxIa8T+CVU1CHOtiir1I+sWLZFy8CJm35jpRzs639YRyipCTlxfKyY6n2E5uMzzOzu0F38aGx1ndB+w2ffNNRIQ/ouKZvvE4oScT8CpXhuc71OapNjUp72LH91ujyYEWGAsozgKTk/kH5/P5ns/pXrM7H3f8GGcHZyQzk4xL8eZ5oNi/54QumHtHsUaPSNJyhNR3cMDJx+c2LznXRo1wCwlB2cMaFRE4tAJ+nwSXT0JAByN9c9VgW1tmEWEnE5i+8Tjbjl/Co6wzz7WrxTNtA/Bwc7a1aRpNvmiBsYCSJDAA30d+zye7PqFr9a581ukznB3zf1iJCJmJibfMCaXH3nRQMIvQ+fOYrhvpd5yrVMFjQH88+/fH2d/f2peUPxlpsGc+bJkM1+OhyQBjsWalWra2zCLCzyQyY2MUGw7H4u7ixNNta/Jcu1p4lbffiAYajRYYCyhpAgOw8PBCJodNpnO1znze+XPKOBZOzvnMq1dJ3raNxGXLSN6xE5SiXLt2eA4ciHvXLnflam0VUq7Ajmmw8yvITIdWw43wM+W8bWuXhRw6l8TMTSf47eB5XJ0cGRpSgxEda+NbwdXWpmk0t6EFxgJKosAALDqyiA9DP6RD1Q5M6TIFF8fCfRtOizlL0k8/kfjTT2RcuIBjxYp49O2L58ABuNSrV6jnumuuXoDN/4a934OzG7R/Be5/Gcq42dYuCzkee5WZm0+wMvwsTo4OPNGqOi92qkNVT/v2mNOULrTAWEBJFRiApceWMmnnJNpVaceXXb7E1anw34QlM5PkHTtIXLacqxs3Qno6ZQMD8Rg4gAq9HsKxfLlCP6fFxB2FDe/D0dVQvrLhcRY0FByLx2T6qUvJfL35BMv3xqAUDAiuxkud61DTy4b3VFNyuHLO+L8o4HyqFhgLKMkCA7Di+Aom7phIiH8I07pOo6yT9d6CMxISSFq5isTly0iLOoFyc6NCrwfxHDiQskFBtnON/mun4XEWEwbeDeCB96BBL7v3OLvJ2cQbfLvlBIt2nSHTJDwSWIVRXepQ19fd1qZpihsZqXBktbGe7MRGeOpnqN2pQE3ZhcAopR4EpmIkHJsjIpNzHHcB/gu0wEhi9riInDIfawZ8C1QATEArEUlRSq0B/DGSpW0DXhaRTKVUIPANUB44BQwVkdtXLGajpAsMwMqolUz4YwKtKrdietfpuDlbd6hIRLgRHk7i8uVc+e1/yPXrlKlTB88BA/B4pC9OXl5WPX8eRsGRX401NPFRUKOt4XFWvVXR21JAYq+kMHtrNAtDT5OSkclDTfwZ3bUuDf118jPNHRCB8/sNUYlYAimJxhqyoMHQ4pkCJ/yzucAopRyBY0B3IAYjO+VgEYnMVmYU0ExERiqlngD6icjj5jTJe4GnRGS/UsoLSDQLSQURuaKMV+JlwFIRWaSU2gW8ISJblFLPAbVEZMKdbCwNAgPwy4lf+L8//o/mvs2Z2W2m1UXmJpnXkrm65n8kLlvOjfBwcHLCvWtXPAcOoFy7doUaCNQyg9Jh739h82RIvggN+0K3ieBdN6vI9fTr7L24l7DzYeyJ3YNfOT+GNxlOY+/GRWtrHsRfS2Xu9pP8d+ffyc/GdK1LoE5+pslO8iVDUMIXQuxBcHSBhr2NYeLane95cbI9CEwb4D0R6WnefhtARP6drcxac5mdZlG5APgAvYAhIvLkHdp3Bn4CFojIYqXUFcBDREQpVR1YKyKN7mRjaREYgP+d/B9vb3ubQJ9AZj4wk3LORTuWnxoVReKy5SStXEnm5cs4Va6MZ/9+ePTvT5lqRZw2OfUa7JwBf0wjNTOFiKZ9Ca3SkLCEQxyIO0CGZODk4EQTryacSDzB1fSrtK3SluebPk9Lv5Z2EQkhZ/KzjvV9GNO1rk5+VprJzICoDRC+AI6uAVM6VGluiErTgVC2YqGdyh4EZiDwoIg8b95+CggRkdHZyhw0l4kxb58AQoAnMYbNfDEEZ5GIfJKt3lqgNfA/jF5OplJqB/CxiKxUSr0OvC8itw1UK6VGACMAatSo0eKvv/6ywtXbJ2tPrWXc1nE08W7CNw98Q/ky5YvcBklL4+qmzYa78/btIEK5tm3wGDAA9wcewMHFuus/MkwZHIo/RNj5MEJjthMeF04qJhxEaOziTas6DxFStT1BvkG4ObtxLe0ai48u5vvI74lPiSfQJ5AXmr5Ax2od7UJorqaks+DP07ckPxvbrR5t6+jkZ6WGuGOGqOxfBNdiwc0bmj0OzYeCn3V63vYgMI8BPXMITGsRGZOtzCFzmewC0xp4FngZaAVcB34H/k9Efs9W1xVYCHwjIuuVUvcB0wAvYBUwVkTuOOBfmnowN9nw1wbe3PImjbwa8XX3r6lQxnZj+OnnzpG4YgVJy38i/dw5HDw88OjTB8/HBuLaoEGhnMMkJo5dPkbo+VDCLhjDXsnpyQDUr1if1pVbE1K+Bi0O/Ir74V+hnC90HgfBwyDbQtWUjBR+jvqZ7w5+x7nkc9SvWJ/nmz5Pj5o97CKS9fW0DH4MO8O3W05w8apOflbiSbkCh36CfQsNBxblCPV6QPMnjU8n665LsweBuZchsscxejbPmMtNAFJE5NMc5xiGMfk/Osf++hhDZ63vZGNpFBiAjac38o8t/6BBxQZ82/1bPFw8bGqPmExc//NPw915/XokPR3XJk3wHDiACg8/jKO75R5TIsLJKycJOx9G2IUwdl3YRWJqIgABFQJoXbk1rf1b06pyKyq55hhOOrPL8Dg7vQO86hrzMw373OJxlm5K538n/8ecA3M4mXSSGu41eK7Jc/Sp06fQFrXeCynpmSzdE8M3OvlZycNkgr+2G6ISuRIybhiekc2fNHos7n5FZoo9CIwTxiR/N+AsxiT/EBE5lK3My0DTbJP8/UVkkFKqIkavpT2QBqwBpgBbAHcROW9ufyGwTURmKKV8ReSiUsoBmA9sFpF5d7KxtAoMwJYzW3ht82vU9azLrO6z8HS1j0nijMuXufLLryQuW0bqsWMoV1cq9OyJ52MDKduiRa5v42evnTWGvC6EEnY+jLgbcQBULleZkMohhPiH0KpyKyqXq5y/ASJwbI3hcRZ3BCoGQO0uUKcL1OqYNY5tEhMbT29k9oHZRMZH4uvmyzONn2FAvQFF5kRxJ9IzTazYe5avNkfxV/x16vuV5+UuOvlZsSTxNIT/aEzYJ/4FLhWMkEjNn4SqLWzicm9zgTEb8RDwJYab8jwR+VApNQnYLSKrzMNc3wPNgQTgCRGJNtd9EngbI6vmbyLyllLKD/gVcDG3uRF4TUQylFKvYAyrgTH5/7bkc3GlWWAAtsVs49VNr1LLoxaze8ymomvhTQLeKyJCysGDJC5bzpVff8WUnEyZgAA8BvQns2cHdmecIOxCGKHnQzl77SwAlVwrEVI5hNb+rQmpHEI192oFHx7KzIADSyByFZzaDmlXQTmAf5AhNrW7QPXWiGMZdp7byZyDc9h1YReeLp4MbTiUwfcNtnnPEIzkZ79GnGfGpiiizMnPRnWuw6M6+Zl9k34DDv8C+xbAya2AQK1Ohqjc19vmkSnsQmDsndIuMAA7zu5g7Kax1KhQg9ndZ+NV1gbrVPLh8uULHF4+l8xV6/A+dpFMBXvrKna2cMOlXRtaVbufkMoh1PGsY535hsx0OLsHTmyC6E0Qsxsk0whFU7NdluCEk8qcg3PZErMFNyc3Hr/vcZ5u9DTeZW0fD81kEtYeMpKfRZ7Xyc/sEhE4uxf2fQ8Hf4LUJCNNeNBQCBwMFWva2sIstMBYgBYYgz/P/8mY38dQtXxV5vScY/MHYnJ6Mnti92TNoxxJOIIglHUqS1fuo3uEwn/bMVRCIk4+Pnj064fngP6UqVlE/4ApSUav5sQmiN4M8ceN/eX9oHZnjvo3Yu71E6w9uxUn5US/ev14tsmzVC1ftWjsuwMiwsYjF5m2MYr9OvmZfXDtouEBFr7QGJZ1KguN+hrCEtChwOFcrIkWGAvQAvM3uy7s4uXfX6ZyucrM7TEXHzefIjt3SkYK++P2Z3l6Hbx0kEzJxNnBmSDfIMPTyz+EJl5NslIQSHo617ZuJXHpMq5t3QomE26tW+M5cADuPXrg4FqEUYgTzxhCE20WnOtGttG/fBvwnZc3K1PPIcDDtR/muSbPUcezTtHZlgciwvaoS0zfGEXYyQS8yxvJz568Xyc/KxIy0+HYWkNUjq01esTVWhmi0qQ/uNp+ePVOaIGxAC0wt7Indg8vbXgJPzc/5vSYg18563ilpJvSOXTpUJaghF8MJ82UhqNypLF346x5lCCfIIuCdKbHxpK04mcSf/qJ9NOncXB3x6NPbzwGDKBs4yJegW8yQewBQ2hObILTO7kg6fzX04Nl7u7cUEI3n2Ceb/E6TfwCi9a2PAiNjmfGpqis5GcPNa1MSC0vQmpXwt9DR3EuVGIjDVHZvwiuXzJ6vc0eN+ZWfArHNb8o0AJjAVpgbmffxX2MXD8Sr7JezOs5zzLPq3zINGVy9PLRLE+vPbF7uJFxA4D7Kt2X1UMJ9g2+p8WfYjJxfdduEpct4+q6dUhqKi4NGxpx0Pr0xtHDBm+F6SlweidEb+Jy9EYW3jjNDxXcueroQBtVnhdq9KRlkydRXnVsHoBz3+nLzN4Wzbbjl7iakgFA9UplDbGpVYn7a3tRrWJZva7mbrmRCAeXGe7F5/aCgxPUfxCaP2Wk+i4mEb6zowXGArTA5E74xXBe2vASHi4ezOs5jyrlq9xVfREhOik6q4ey68IurqQZcUdredTKEpSWfi2t5rmWeeUKSb+a3Z0jD6PKlMG9Rw88Bw7ErXUr26V9To7nWtQalhxZzH+vRxPvoAhMSeWFDBc6VuuMqtPFiBXlZruQL5km4fD5K4SeTCA0Op6wUwkkXk8HoIqHKyG1DcEJqe1FgJebFpzcMGXCyS2GqBz+BTJTwbexec3KoGKTCC8vtMBYgBaYvDkQd4AX17+Iexl35vacSzX3vOOFiQgx12JuWYsSn2LMQ1QtXzVrcWPryq3xdfMtqkvIIiUy0oiD9ssvmK5exbl6dTwH9MejXz+c/YpucdptdqXf4OeIOXx39EfOpV+lfnomwy9fpkfyDZz8A/92h65xPzjZLoWyySQcv3iN0JPxhEYnEHoynkvX0gDwdXehtVls7q9Vibq+5Uu34CREQ/gPxrqVKzHGXErTQUbYFv8gm/dSCwstMBagBebOHIo/xIh1IyjnXI65PeZSvUL1rGOxybGEXTC8vMLOh3Eu+RwA3mW9CfEPIaSysbjxTsJU1JhSUri6fj2Jy5ZzPTQUHBwo36EDHgMH4N65M8rZOf9GrEC6KZ01J9cw58BsopNOUt2pPM+lOdH3TCRlTBmGV1HNNn8v+PRtbFPPIhHhRFzyLYITeyUVAK9yZQzBMYtOAz/3kh9BIC3ZWFm/b6Gx0h4FdboaotLgYXAueWmvtcBYgBaY/Dkcf5gX1r+Aq6MrY5qP4cClA4SeD+XUlVMAeLh4GD0Ucy+lVoVaxeINNu30aRKX/14b4mkAAB0GSURBVETSihVkXLyIo5cXHo8+gueAgbjUrmUTm0xiYtPpTcw6MMuIDlDWm2G+bRh4PRW36G1w6ahRsJyPMYxWu7MhOh62dX8WEf6Kv07YyQT+NIvO2URjjs2jrDOtAipxf+1KhNTyolGVCiUjkoAInAk1FkIe+v/2zjy66irb8599782cECABQQYRhBIZHAAHHAqKUlARxQlBELqUwbJ8b9V7VXbbXd2r2lqvrWe9bl/3qnqCMgsKVaBCWyAOOPCqUUQFmZUxCYNAEjJP997df/x+JDchJD8gNzfD/qx1V37DOee3Dz/u+d4z7f02VBZDpysdUbl28kXHWWktmMB4wATGG/vy9jHz/ZnkV+STHEhmeLfh1fMoAzoNwCctb52+VzQYpHjTJs6sXk3xJ59CMEjSsGF0fOghOowbiy+5+XdMqyqbj29m/o463gF6jCY956ua/TclJ50MmQNqejd9boOE2Ee7zMkvre7dfHEojyO5pQCkJQQY3qdT9TzO4B7prcujQOFx2O66bcndD3EpMOgBZ26l9y1tZgisMUxgPGAC453cslyOFR/j6oyrifPFZigp2gRPnaJgzRrOrFpN5eHD+FJSSJ84kYwnf0Zc9+4xsWnbyW3M3zG/xjvAjyYx7ZppdEnKhB921ey/Ofw3x/mhLwA9htfM3/QY1iJWKZ0oKOeLQ7l8fjCPLYdyOXDK8WidHO9n2BWdqofUhvZMb3meBYIVsG+9Iyr7PwQNO2Jy/VS45v4WIejNjQmMB0xgjPpQVcq+/pozf/4zBX9dByJ0fOABMmbNJL5Xr8YLiAL78vaxYOcCNhzeUO0dYMagGTVzXMEKZ8jmrDubY9sAdRwj9rndFZxRjpfoFvAr+1RRBVsO5VXP4+z7oQiAhICPG3p34iZ3SO363h1JjIuR4Bz/1g01vBLK8iHtcifU8HWPQ0bsN8vGEhMYD5jAGI1RdfQouQsWcGbVajQUIn38vWTMmkVCv9g0MFmFWSzcuZA1B9agqtxz5T08OeTJc70DlOY5ThIPfuyIzhk3sF6HntBvlNO76TuqxSyXzSuprCU4e04Uogrxfh/X9kqv3vg57IpOJMdHsUdWmueGGl4GJ3aAPx6uvheum+qIdAuI/dMSMIHxgAmM4ZWqkyfJW7SY/BUr0PJy0saOJXP2LBIHDoyJPSdKTrB091JWfbeKsmAZY3qP4akhTzE4c3D9GfIO1ngXOPSp408NoNtQR2j6jXaGfeJaxs79grIqth7Oq96Ls/NYIaGwEvAJQ3rWCM7wKzqRlniJQ7bhEBzY6DiZ3LceQpXQ/VpHVIY8HNM9SS0VExgPmMAYF0owP5+8JUvIX7accHExqaNGkfn0HJKujY3bl/zyfN7Y+wbL9yynqLKIm7vfzMwhMxnRbcT5V/OFQ84Q2sGNcOATZ2gtXAWBRGfPTWo3QJ2VUmf/avg812gkXbj2/ep0da/Vl67mWiislFZUUVZZRVllkPKqEKgiKElxQlKcj6Q4P4kBwS/UKY+Gn1FV7oRjSOpcE2q425CovK+2ggmMB0xgjIslVFhI/vLl5C1eQqiggJSRt5AxZw7JIxpo2KNIcWUxf/nuLyzZtYTc8lyGdhnKzCEzuaPnHY2v8qssgSP/z+ndHP7M7d2IM1cjvprjBq+Jx3RNkzcUhvyyKvJKq8gtqSK/tIqQgiKkJcWTkZJARloimakJJMQFIvJybnk+vxNMbsDdUQ813FZoEQIjIuOA/40THGy+qv6+zv0EYCkwDMgFJqnqYffeUGAe0AEI44RGLheR94DuQADYBDyjqiERuQ6YCyQCQeDnqrqlIftMYIxLJVxSQv6KleQuWkTo9GmShg0jc84cUm67NSZCUxGq4J3v32HRrkUcLT5K/079eWrwU9zV5y4CvtivJosW5VUhtmWfqV4a/XVWPuVVTg+rf9fU6kUDN/XtTNe0lrnxMRRWyqtCzicYrjmuClFeFa79Nxh5rU6aiLwVVWHKgyHKKmvnqagKs2DGcG7vf3Fe02MuMCLixwmZfCeQgxMyebKq7o5I83NgaETI5ImqOskNh/w1ME1Vt4tIBnDGFZIOqloozrd3FfAXVV0hIu8DL6vqejeS5nOqOqohG01gjKYiXF7OmdWryZ2/gODx4yQOHkzm03NIHT06Jn7ParwDzOdgwUF6pfXiZ4N/xoR+E4j3t/1f6ZXBMDuOnuHzg848zleH8yipDAHQNzOFm/p2dj0OZHB5x/rnnYKhcJ2G3m2cIxrqsqraDX1FnfRlEcdOvtoiUVYZpsI9rgpdfFucGOcjMc5PYsBPUryfhIB7HnG9+tj9PDK8J/26XJxz2ZYgMLcAv1XVse758wCq+mJEmg1ums2uqJwAugB3A1NUdWoD5cfhhEZepqor3bIWuseTgftUdUpDNprAGE2NVlZSsHYtp199jaqsLBL69ydjzmw6jBuH+Jt/BVJYw3yc/TGvffsau3J30TWpK08MeoJHBjxCclxsw+42J8FQmJ3HCvnioLPx88tDeRRVOB6je3RMIineXyMEl9jgi3BOg362wU+Kq9vQ+0gI1BwnRVx38tVOn1RHNBLifCQEfM3eW24JAvMwME5Vn3LPpwE3qeovItLsdNPkuOcHgJuAqTjDZl1xBGeFqr4UkW8DcCOwHqeXExKRgcAGnFFWHzBSVY80ZKMJjBEtNBikcP16Ts+dR+WBA8T36UPGrFmk3zc+Jj7PVJXPj3/O/B3z2XJiC+kJ6Tw+8HGmXD2F9ISWHdwqGkR6jP46K59wWEmK85Nw3l/9tX/9JwZqi0Tkcby/+Rv85qYlCMwjwNg6AnOjqj4bkWaXmyZSYG4E/gPwDDACKAU+An6jqh9F5E0ElgNzVfUDEfk/wKequlpEHgVmqepP67FrFjALoHfv3sOOHGlQgwzjktBwmKIPPuT03LlU7NlD3OWXkzFrJukTJ+JLiI2H5G0nt7FgxwI+yfmktneAZoxiarRuWoLAXMoQ2SScns0MN91/BcpV9Q91njEdZ/L/FyJSAHRUVXXnZwpUtUNDNloPxmguVJXiTz8l95W5lG3fTqBrVzKe/BkdH3kkJv7OwIN3AMM4Dy1BYAI4k/xjgKM4k/xTVHVXRJpngCERk/wPquqjItIJp9dyG1AJvAe8DHwKpKnqcbf85cAmVf2jiOwBnlbVT0RkDPCSqg5ryEYTGKO5UVVKv/iC06/MpfSLL/B37kzn6dPp9PgU/KkXH83zUsgqzGLRrkWs2b+GsIYZ0GkAvdJ60btDb3qn9aZXWi96pfWiS3KXVu3Y1Gg6Yi4wrhH3AP+Ks0x5oar+k4i8AGxV1bXuMNfrwPVAHvCYqh50804FnsfZJrVOVZ8TkcuAd4EEt8yNwC9VNSgit+EsiQ4A5TjLlL9qyD4TGCOWlH79NafnzqXks034OnSg89SpdH5iGv6OHWNizw8lP7By30r25O0huyibo0VHCWqw+n6iP5GeaT3pndab3h1qhKd3h950S+6G39yotBtahMC0dExgjJZA2c5d5M6bS9EHH+JLTqbTlMl0njGDQGZs/YQFw0FOlJwgqyiL7MJssoqyqo9zinOoCFVUpw34AvRM7Vnd64kUostTL2+zHrjbKyYwHjCBMVoS5d99R+68Vylcvx6Ji6Pjo486oQK6dYu1aecQ1jAnS0+SXZRNVqErPO5xdlE2pcHS6rR+8dM9pXvtXo8rPj3TepLgj104aOPiMIHxgAmM0RKpOHSI3NfmU7B2rRMqYOJEMmY+FbNQAReKqpJbnltbfAqzyS7K5kjREYoqi6rTCsJlKZdVi07duZ/2tFenNWEC4wETGKMlU5lzlNwF8ylYtRoNh51QAbNnk9C3b6xNuyQKKgqqhSdSfLKKssgrz6uVNiMxo7rnU3fupz3u32kpmMB4wATGaA1U/XCSvEWLyF+50gkVMG4smbNnk3j11bE2rckpriyuFpvsouxavaCTpSdrpU1PSK/V64nsBXVO7NzmNzvGEhMYD5jAGK2JYF4eeUuWkr9sGeGSElJHj3ZCBQwdGmvTmoWyYBk5RTlkFWU5fyPmfo6XHCd8NnwAkBKXUmuJdaQA2XLrS8cExgMmMEZrJFRQQN7y5eQvWeqGChhJ5tNOqID2SmWokqPFR8/p9TS03Dpy2G10r9HmyeACMIHxgAmM0ZoJFZdwZuUKchcuIpSbS9LwYWTOeZqUW0fa8FAEwXCQ4yXHHfGps9w6uyibynAlcb44xvcdzxPXPMFVna6KtcktHhMYD5jAGG2BcHk5Z1atJnf+fIInTpA4ZIgTKmDUqJiECmhNhDXM4YLDvLn3Td7Z/w7loXJu63EbMwbN4MZuN5pQnwcTGA+YwBhtCa2s5MyaNeS++hpV2dkkDBhA5pzZpI0dG5NQAa2NM+VnWLlvJW/sfYO88jwGdh7I9EHTuavPXbZRtA4mMB4wgTHaIhoMUrhunRMq4OBBJ1TA7Nmkj783JqECWhsVoQrePfAuS3Yv4VDBIbqldGPqwKk81P8hUuNj4y+upWEC4wETGKMto+EwRe9/4IQK2LuXuB49yJg5k/QHJ+KLb/tRLS+VsIbZlLOJJbuX8OWJL0mNS+XhAQ/z+MDH6ZbS8rwrNCcmMB4wgTHaA6pK8SefcHruXMq3f+uECnjqSSdUQFL94YKN2uw6vYslu5bw/pH3EYRxV45j+qDpXN257e1F8oIJjAdMYIz2hKpS+vnnTqiALVucUAEzZtBpyuSYhQpobRwtPsqy3ct46/u3KA2WcnP3m5k+aDq3Xn5ru1oQYALjARMYo71S+tVXnJ47j5JNbqiAadPoPG1qzEIFgCOABINoVRV69m/1p+acYFXte7XS1skXkZZgEK2sJ18wiC8xEV9aGr7UFPypqfhSI47T0vClpOJPTXGPUygKlbDqu1Us372ck2UnuarjVUwfNJ17rryHeH/bH340gfGACYzR3inbsZPT8+ZS/OFH1aECEgcPiWiAK2sa5/M0+OeIgduoU2+Df/58VFVFr6IiSFxc9Ye4QM25P0C4vIxwcQnh4mIIhxsvLjkZf2oqkppCYVyQo+E8TvtKCack0qf7QAb0vJ7kjpnnF6zUVHzJya12GXmLEBgRGYcTBMwPzFfV39e5nwAsBYYBucAkVT3s3hsKzAM6AGGc0MjlIvIe0B0nsNgm4BlVDYnISuBHbtEdgTOqel1D9pnAGIZD+b7vyJ03j8L33vPWwFY31DWNtgQCtRrxyPNaDXpcHBKIqz9tfM0x9aZt4BmBABIXf045joh4W6atqmhpKaHiEsLFRYSLiwkVFxMuKiZcUkyoqKhaiELFEcdFRZScOUVpwWn8pZUkV3p4mAi+lBR8qan401LxpaQ6wpOW6ghRSsTx+XpVaalIUlKzD8/FXGBExI8TMvlOIAcnZPJkVd0dkebnwNCIkMkTVXWSGw75a2Caqm4XkQwcwQiJSAdVLRTnX3QV8BdVXVHn2f8TKFDVFxqy0QTGMGpT9cMPhM4U1N9Iuw06gUC7mm+4UPbl7WPpziV8vG8diRVhfpJxCw90v4s+/q61BauOSIWLi1xhKyZcVESopAQtLW38gT6fI1LVQuQKU72CVSNSiQMHXvSQaEsQmFuA36rqWPf8eQBVfTEizQY3zWZXVE4AXYC7gSmqOrWB8uOAt4Blqroy4roAWcBPVPX7hmw0gTEMI1qcKDnBG3vfYNW+VRRVFTHssmHMGDSDO3re4dnZpgaDhEvO9pgurFd1VqTCxcVoefk5Zfd67VVSb7/9ourmVWACF1W6N3oA2RHnOcBN50ujqkERKQAygAGAugLUBVihqi+dzeRevxFYj9OLieR24IfGxMUwDCOadEvpxj8M+wdmD53NW9+/xeu7X+fZjc/Sp0Mfnhj0BPf1vY/EQGKDZUgggD89HX96OpeyRVYrKx2xKSkhXOSIVEL//pdQojeiOcNUXx+6bnfpfGkCwG3A4+7fiSIypjqB0yvqDiQAP6mTfzLw5nmNEpklIltFZOupU6carYRhGMalkBKXwrRrprHuwXW8dMdLJMcl88LmFxi7eiyvbH+F/PL8qNsg8fEEOnUivmdPEgcOJHnEiGZZMRhNgckBImO89gSOnS+NO0SWDuS51z9V1dOqWgqsA26IzKiq5cBa4P6z19wyHgRWch5U9VVVHa6qw7t0MffchmE0DwFfgLuvvJsV965g4diFDMkcwr9t+zfuXHUnv9v8Ow4XHI61iU1ONAXmS6C/iFwpIvHAYziCEMlaYLp7/DCwUZ1JoQ3AUBFJdkXjx8BuEUkVke5QLSb3AHsjyvspsFdVc6JWK8MwjEtARBjRbQR/HPNH1ty/hvF9x/P2/reZ8M4E/n7j3/PNyW9oK9tHor1M+R7gX3GWKS9U1X8SkReAraq6VkQSgdeB63F6Lo+p6kE371TgeZwhs3Wq+pyIXAa8izM05gc2Ar9UdaIJichi4HNVnevFPpvkNwyjJXC67DRv7n2TlftWUlBRwNDMoUwfNJ0xvcfg97U8T9gxX0XWGjCBMQyjJVFaVcraA2tZunsp2UXZ9EztybRrpvHAVQ+QHJcca/OqMYHxgAmMYRgtkVA4xMfZH7N412K2n9pOh/gOTPrRJKYMnEJmUmaszTOB8YIJjGEYLZ1tJ7exZNcSPsr6iIAvwPi+45k+aDr9OvaLmU0tYR+MYRiGcYlc1/U6rut6HVmFWSzdvZQ1+9fw9v63ub3H7cwYNIMR3Ua0WM8K1oOxHoxhGK2I/PJ8Vu5byZt734xZaGcbIvOACYxhGK2VWIZ2NoHxgAmMYRitnbOhnRfvWszWH7Y2S2hnExgPmMAYhtGWaK7QziYwHjCBMQyjLXI2tPPq71dTFizj5u43M2PQDEZePrJJFgSYwHjABMYwjLZMQUUBq75bxRt73mjS0M4mMB4wgTEMoz1QFapi/eH1LN61mO/zv6dLUhdevP1FbupeN4KKN2wfjGEYhgFAnD+OCf0mcF/f+9h8bDNL9yzlig5XRP25JjCGYRjtBBFhZI+RjOwxslmeF013/YZhGEY7xgTGMAzDiAomMIZhGEZUiKrAiMg4EdknIvtF5D/Vcz9BRFa6978QkT4R94aKyGYR2SUiO9zgZIjIeyKy3b0+V0T8EXmedZ+3S0ReimbdDMMwjIaJ2iS/2/D/CbgTyAG+FJG1qro7ItmTQL6qXiUijwH/DExywyEvA6ap6nYRyQCq3DyPqmqhOLuFVgGPACtEZDRwPzBUVStEpGu06mYYhmE0TjR7MDcC+1X1oKpWAitwBCCS+4El7vEqYIwrHHcB36rqdgBVzVXVkHtc6KYPAPE4IZUBngZ+r6oVbrqT0amWYRiG4YVoCkwPIDviPMe9Vm8aVQ0CBUAGMABQEdkgIl+LyHORmURkA3ASKMIRJtw8t7tDbZ+KyIimrpBhGIbhnWgKTH0Ob+q6DThfmgBwG/C4+3eiiIypTqA6FugOJAA/cS8HgE7AzcCvgT9LPU53RGSWiGwVka2nTp26sBoZhmEYnonmRsscoFfEeU/g2HnS5LjzLulAnnv9U1U9DSAi64AbgI/OZlTVchFZizPM9oGb5y11fN9sEZEwkAnUUhFVfRV41S33lIgcucj6ZQKnLzJva8Xq3D6wOrcPLqXOntwARFNgvgT6i8iVwFHgMWBKnTRrgenAZuBhYKOqqjsE9pyIJAOVwI+Bl0UkFUhT1eOuIN0DbHLLegenN/OJiAzAmZ9p8B9PVbtcbOVEZKsXXzxtCatz+8Dq3D5ojjpHTWBUNSgivwA2AH5goaruEpEXgK2quhZYALwuIvtxei6PuXnzReR/4YiUAutU9a8ichmwVkQS3DI3AnPdRy4EForIThxRmq7t2ZOnYRhGjGnX3pQvBfvF0z6wOrcPrM7RwXbyXzyvxtqAGGB1bh9YndsHUa+z9WAMwzCMqGA9GMMwDCMqmMA0ggd/anNcX2nbROTfReSaWNjZlDRW54h0D4uIikirH7v28J5nuMvat7mfp2JhZ1Ph5R2LyKMistv17fdGc9vY1Hh4xy9HvN/vRORMLOxsSjzUubeIfCwi34jItyJyT5MaoKr2Oc8HZ6XaAaAvzrLn7cA1ddJ0iDieALwXa7ujXWc3XRrwGfA5MDzWdjfDe54B/DHWtjZjffsD3wCd3POusbY72nWuk/5ZnJWvMbc9yu/5VeBp9/ga4HBT2mA9mIZp1J+a1vhGA0jhXG8FrQ0vPuQAfge8BJQ3p3FRwmud2wpe6jsT+JOq5kOb8O13oe94MvBms1gWPbzUWYEO7nE6526GvyRMYBrGiz81ROQZETmA0+D+XTPZFi0arbOIXA/0UtV3m9OwKOLpPQMPucMIq0SkVz33Wwte6jsAGCAifxORz0VkXLNZFx28vmNE5ArgSpx9dq0ZL3X+LTBVRHKAdTg9tybDBKZhvPhTQ1X/pKr9gP8I/CbqVkWXBussIj7gZeAfm82i6OPlPf9foI+qDgU+pMYLeGvES30DOMNko3B+zc8XkY5RtiuaePouuzwGrFLXg3srxkudJwOLVbUnjmeU193veJNgAtMwXvypRbICeCCqFkWfxuqcBgzGcclzGMe56NpWPtHf6HtWJ2REhXv6GjCsmWyLBl79BK5R1SpVPQTswxGc1sqFfJcfo/UPj4G3Oj8J/BlAVTcDiTg+ypoEE5iGqfanJiLxOP/x1kYmEJHIL929wPfNaF80aLDOqlqgqpmq2kdV++BM8k9Q1a2xMbdJ8PKeu0ecTgD2NKN9TU2j9cXx7TcaQEQycYbMDjarlU2LlzojIj/C8cq+uZntiwZe6pwFjAEQkYE4AtNkbuaj6eyy1aPe/Kn9QkR+ihNxMx/HeWerxWOd2xQe6/x3IjIBCOL4zZsRM4MvEY/13QDcJSK7gRDwa1XNjZ3Vl8YF/L+eDKxQd1lVa8Zjnf8ReE1EfokzfDajKetuO/kNwzCMqGBDZIZhGEZUMIExDMMwooIJjGEYhhEVTGAMwzCMqGACYxiGYUQFExjDiBIi8lsR+VULsOOwu5fFMJoVExjDMAwjKpjAGMYFICIpIvJXEdkuIjtFZFJkD0FEhovIJxFZrhWRjSLyvYjMdNN0F5HP3LgjO0Xkdvf6KyKy1Y2/8t8jnnlYRP6HiGx2798gIhtE5ICIzHHTjHLLfNuN4TK3Pp9SIjJVRLa4z54nIv5o/nsZ7RsTGMO4MMYBx1T1WlUdDLzXSPqhOC6EbgH+m4hcDkwBNqjqdcC1wDY37X9R1eFunh+LyNCIcrJV9RZgE7AYeBjHD9wLEWluxNmZPQToBzwYaYjrCmQScKv77BDw+AXU3TAuCHMVYxgXxg7gX0Tkn4F3VXWTSH1Oa6tZo6plQJmIfIwjAl8CC0UkDnhHVc8KzKMiMgvne9kdJwDUt+69s65MdgCpqloEFIlIeYSX4y2qehBARN4EbgNWRdgyBsdJ55euzUlAa4/zYrRgTGAM4wJQ1e9EZBiOa/MXReR9HP9kZ0cDEutmObcI/UxE7sDp2bwuIn/A6Zn8ChihqvkisrhOWWc9OYcjjs+en/0en/OsOucCLFHV5xuppmE0CTZEZhgXgDvEVaqqy4B/AW4ADlPjvv+hOlnuF5FEEcnAia3ypRvQ6qSqvgYscMvoAJQABSJyGXD3RZh3o+s514czFPbvde5/BDwsIl3dunR2bTGMqGA9GMO4MIYAfxCRMI4H7adxhpoWiMh/Br6ok34L8FegN/A7VT0mItOBX4tIFVAMPKGqh0TkG2AXjlv8v12EbZuB37s2fga8HXlTVXeLyG+A910RqgKeAY5cxLMMo1HMm7JhtAFEZBTwK1UdH2tbDOMsNkRmGIZhRAXrwRiGYRhRwXowhmEYRlQwgTEMwzCiggmMYRiGERVMYAzDMIyoYAJjGIZhRAUTGMMwDCMq/H8IIMjjNnutHQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch3_1.best_score_, gsearch3_1.best_params_))\n",
    "test_means = gsearch3_1.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch3_1.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch3_1.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch3_1.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch3_1.cv_results_).to_csv('my_preds_subsampleh_colsample_bytree_1.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(colsample_bytree), len(subsample))\n",
    "train_scores = np.array(train_means).reshape(len(colsample_bytree), len(subsample))\n",
    "\n",
    "for i, value in enumerate(colsample_bytree):\n",
    "    pyplot.plot(subsample, -test_scores[i], label= 'test_colsample_bytree:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'subsample' )                                                                                                      \n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig( 'subsample_vs_colsample_bytree1.png' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'colsample_bytree': [0.75, 0.8, 0.85], 'subsample': [0.75, 0.8, 0.85]}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#max_depth 建议3-10， min_child_weight=1／sqrt(ratio_rare_event) =5.5\n",
    "subsample = [i/20.0 for i in range(15,18)]\n",
    "colsample_bytree = [i/20.0 for i in range(15,18)]\n",
    "param_test3_2 = dict(subsample=subsample, colsample_bytree=colsample_bytree)\n",
    "param_test3_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/sklearn/model_selection/_search.py:762: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.06377, std: 0.00102, params: {'subsample': 0.75, 'colsample_bytree': 0.75},\n",
       "  mean: -0.06382, std: 0.00109, params: {'subsample': 0.8, 'colsample_bytree': 0.75},\n",
       "  mean: -0.06374, std: 0.00108, params: {'subsample': 0.85, 'colsample_bytree': 0.75},\n",
       "  mean: -0.06367, std: 0.00105, params: {'subsample': 0.75, 'colsample_bytree': 0.8},\n",
       "  mean: -0.06364, std: 0.00116, params: {'subsample': 0.8, 'colsample_bytree': 0.8},\n",
       "  mean: -0.06368, std: 0.00106, params: {'subsample': 0.85, 'colsample_bytree': 0.8},\n",
       "  mean: -0.06371, std: 0.00114, params: {'subsample': 0.75, 'colsample_bytree': 0.85},\n",
       "  mean: -0.06366, std: 0.00108, params: {'subsample': 0.8, 'colsample_bytree': 0.85},\n",
       "  mean: -0.06367, std: 0.00114, params: {'subsample': 0.85, 'colsample_bytree': 0.85}],\n",
       " {'colsample_bytree': 0.8, 'subsample': 0.8},\n",
       " -0.06363784022623405)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb3_2 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=86,  #第二轮参数调整得到的n_estimators最优值\n",
    "        max_depth=5,\n",
    "        min_child_weight=2,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch3_2 = GridSearchCV(xgb3_2, param_grid = param_test3_2, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch3_2.fit(X_train , y_train)\n",
    "\n",
    "gsearch3_2.grid_scores_, gsearch3_2.best_params_,     gsearch3_2.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.063638 using {'subsample': 0.8, 'colsample_bytree': 0.8}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaUAAAEKCAYAAACymEqVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzsvXl8VdXV//9eSUhCmJMwhJAEEFAQIlPACZAhAYeKKIJondpKrbXV9lettrW1Pu23Wvs4VNtS6lSnguDEU5DkAoITSsIogzJJSABDSMJMxrt+f5yTcBNuQoDcjOv9et1X7t1373PWuYT7ydr7c9YWVcUwDMMwGgNBDR2AYRiGYZRjomQYhmE0GkyUDMMwjEaDiZJhGIbRaDBRMgzDMBoNJkqGYRhGo8FEyTAMw2g0mCgZhmEYjQYTJcMwDKPRENLQATQ1oqOjtWfPng0dhmEYRpMhOjqa1NTUVFWddLq+JkpnSM+ePcnIyGjoMAzDMJoUIhJdm34Bnb4TkUki8rWIbBeRh/y8HyYic933vxCRnj7vJYrIShHZJCJfiki4275YRNa77bNEJNhtHywin4vIOhHJEJERbvstIrLBfXwmIhf5nGOXe+x1ImJKYxiG0cAETJRcsfgbcCUwAJghIgOqdPs+UKCqfYCngSfcsSHA68DdqnohcAVQ4o6ZpqoXAQOBzsCNbvufgd+r6mDgt+5rgG+AMaqaCPwPMLtKDGNVdbCqDj/3qzYMwzDOhUBmSiOA7aq6U1WLgTnA5Cp9JgP/dp/PB8aLiAApwAZVXQ+gqnmqWuY+P+z2DwFCgfIy5wq0d593APa6/T9T1QK3/XOgR91domEYhlGXBFKUYoEsn9fZbpvfPqpaChwCooB+gIpIqoisEZEHfQeJSCqwHziCI2YA9wNPikgW8BfgYT8xfR/4wOe1AmkislpEZp75JRqGYRh1SSBFSfy0Vd28qbo+IcDlwC3uzykiMr6ig+pEIAYIA8a5zT8CfqaqccDPgBcrnUhkLI4o/dKn+TJVHYozxfhjERnt90JEZrrrVBm5ubn+uhiGYRh1QCBFKRuI83ndA3dKzV8fdx2pA5Dvtq9Q1QOqehxYBAz1HaiqhcACTk4J3g684z6fhzN9iHvsROAFYLKq5vkco3yKbz/wru+YKueararDVXV4586da3XxhmEYxpkTSFFKB/qKSC8RCQVuwhERXxbgiAnAVGCZOlvhpgKJIhLhitUYYLOItBWRGKgQsauAr9zxe91+4GRP29x+8Thidauqbi0/sYi0EZF25c9x1rE21tnVG4ZhGGdMwO5TUtVSEbkXR2CCgZdUdZOIPAZkqOoCnCm210RkO06GdJM7tkBEnsIRNgUWqepCEekKLBCRMPeYy4BZ7invAp51xaoQKF8j+i3OOtXfHQ8Fpa7TrivwrtsWArypqosD9XkYRqAoKi1j6Zb9lHqVqwZ2IyTYCrUYTRdxEhOjtgwfPlzt5lmjMfD1t0eYm57Fu2uzKTju3DHRp0tbfjnpAib074L7B5dhNApEZHVtbr2xig6G0YQ4UljC/63fx9yMLNZnHaRVsJA8oCvThsdRWFLGnxd/zV2vZjCiZyQPX3UBQ+I7NXTIhnFGmCgZRiNHVUnfVcDc9CwWfbmPEyVl9O3Slt9c3Z8pQ2KJahtW0Xd8/67MTc/imSVbmfL3z7hqUDcemHgBvaLbNOAVGEbtsem7M8Sm74z6Yv+RQt5Zs4e30rPYeeAYbUKD+c5F3ZmWFMeQuI41Ts8dLSrlXx/t5F8f76S41MvNI+P56fi+RPsImGHUJ7WdvjNROkNMlIxAUlrmZfnXuczNyGLZV/sp8yrDEzoxLSmOqwfF0CbszCY39h8p5Nkl25iTnkV4SBA/HHMePxjVi4hQmyQx6hcTpQBhomQEgm8OHOOtjCzeXp3N/iNFRLcN5YahPbhxeBx9urQ95+PvyD3Knxd/ReqmHLq0C+Nnyf24cVgPc+oZ9YaJUoAwUTLqihPFZXywcR9z0rNY9U0+QQJXnN+F6UlxjLugC60CIBgZu/L50wdfsTqzwJx6Rr1iohQgTJSMc0FV+XLPIeamZ7Fg3V6OFJWSEBXBtOFx3DC0B906hNdLDKmbcvjz4q/YeeCYOfWMesFEKUCYKBlnw8Hjxby7dg9z07P46tsjhIUEcdWgGKYNj2Nkr0iCguo/Uykp87pOvW0cOFpkTj0joJgoBQgTJaO2eL3KZzvymJuRRerGbyku8zIotgPTkuK49qLudGjdqqFDBOBYUSn/+ngnsz8yp54ROEyUAoSJknE69h48wbyMbOatziK74AQdWrfiusGOlfvC7h0aOrxqMaeeEUhMlAKEiZLhj6LSMpZs3s/cjCw+3paLKlzWJ4ppw+OYeGE3wlsFN3SItcbXqde5XRg/m9CPacPNqWecGyZKAcJEyfClvP7ce+v2kH+smJgO4dw4zLFyx0VGNHR454SvU++8zm146Mr+5tQzzhoTpQBhomQcKSzhvxv2MTc9i3Vu/bkJ/bsyLSmO0X07E9wApoVAoaqkbc7hiQ8cp15Sz048fFV/hppTzzhDTJQChIlSy0RVych06s8t3HCy/tz0pLhT6s81R8ypZ5wrJkoBwkSpZZF7pIi312TzVkYWO3PPrP5cc8ScesbZYqIUIEyUmj+lZV5WbM1lbrpTf67UqwxL6MT04XFcnXjm9eeaI/uPFPLXpdv4zypz6hm1w0QpQJgoNV92ldefW5NNzmGn/tz1Q3swbXgP+nRp19DhNUrMqWfUFhOlAGGi1Lworz83Nz2LL3zqz00bHsf4/oGpP9ccWZ2Zz/9bdNKp98tJF5A8oGuLm940qqdRiJKITAKeBYKBF1T18SrvhwGvAsOAPGC6qu5y30sE/gm0B7xAkqoWishiIAZng8KPgR+rapmIDAZmAeFAKXCPqq4S53/Fs8BVwHHgDlVd457jduA3bjh/UNV/n+6aTJSaPqrKxj2HmZuxm/fX7eVIYf3Xn2uOVDj1Fn/Fzlxz6hmVaXBREpFgYCuQDGQD6cAMVd3s0+ceIFFV7xaRm4ApqjpdREKANcCtqrpeRKKAg674tFfVw67YzAfmqeocEUkDnlbVD0TkKuBBVb3Cff4THFEaCTyrqiNFJBLIAIYDCqwGhqlqQU3XZaLUdDl4vJj31u5hbkY2W/YdJiwkiCsHdmNaUhwX94pqkPpzzZHSMi9zM7J42uM49a4c2I0HJp5P787nvgWH0XSprSgFclVyBLBdVXe6Ac0BJgObffpMBh51n88HnnfFJgXYoKrrAVQ1r3yAqh72iT0UR1Bwf7Z3n3cA9vqc41V11PdzEekoIjHAFYBHVfPd+DzAJOA/53zlRqOhUv25Td9SXOplYGx7/mfyhVw7OLbR1J9rToQEB3HLyASuGxxb4dTzbM5hxgjHqde5nTn1jOoJpCjFAlk+r7NxMhW/fVS1VEQOAVFAP0BFJBXoDMxR1T+XD3LbRwAf4IgZwP1Aqoj8BQgCLq0hjtga2o1mwN6DJ5i/2rFyZxecoH14CDOS4hp9/bnmRJuwEO6f0I9bRibw7NKtvLlqN++syWbmaMepZy5Gwx+B/K3wNxdSda6wuj4hwOVAEs460FI39VsKoKoTRSQceAMYB3iAHwE/U9W3RWQa8CIwoYZz1CY+J0iRmcBMgPj4eH9djEZAcamXJVtymJuexUdu/blLz4vigYnnN7n6c82Jzu3C+MN1g7jzsl48ufhrnl6ylde/yDSnnuGXQIpSNhDn87oHJ6fUqvbJdteROgD5bvsKVT0AICKLgKHA0vKBrulhAc70nAe4HbjPfXse8MJp4sjGmcLzbV/u70JUdTYwG5w1pRqv2qh3tuY49efeXevUn+vWPpx7x/bhxmFxxEc17fpzzYnzOrdl1q3DWJ2Zz58WfcWv3v2SFz/ZaU49oxKBFKV0oK+I9AL2ADcBN1fpswBHTFYCU4Flqlo+bfegiEQAxcAY4GkRaQu0U9V9rohdhePAA0doxuAIyzhgm8857nXXtEYCh9zxqcD/E5Fya1AK8HCdfgJGwDhaVMp/1+9lbkYWa3c37/pzzY1hCZHMu/uSCqfezNdWm1PPqCBgouSuEd0LpOJYwl9S1U0i8hiQoaoLcKbYXhOR7TgZ0k3u2AIReQpH2BRYpKoLRaQrsMC1kgcDy3Bs4AB3Ac+6YlWIO90GLMIRr+04U4F3uufIF5H/cc8B8Fi56cFonKgqq8vrz325j+PFZfTp0pZfX9WfKUNjrdRNE0JEmHhhN8Zf0KXCqXf93z8zp55hN8+eKWYJr39yjxTxjlt/bodbf+6aRKf+3ND4lld/rjlyrKiUFz7+hn9+tIPiUq859ZohDX6fUnPFRKl+KC3z8tE2p/7c0i1Wf66lkHukiGeXbq2oqWdOveaDiVKAMFEKLJl5Tv25+aud+nNRbUK5YZjVn2tp7Mg9ypOLv2bxpm/p3C6M+yf0ZfrwOHPqNWFMlAKEiVLdU1hysv7c5zud+nNj+nVmelIc4y7oSmiIfRG1VMqdehlWU6/JY6IUIEyU6o6New5VbCV+pLCU+MgIpg3vwQ3DehDToXVDh2c0EqymXvPARClAmCidG4eOl/Deuj3MTc9is9WfM84Aq6nXtDFRChAmSmeO16us3JnH3PQsFrv15y7s3p6bkuK49qJYOkRY/Tmj9vg69YpKvdxsTr0mgYlSgDBRqj37Dp1gfkY2b63OIivfqT933ZBYpg2PY2Cs1Z8zzo3cI0X8dek23ly125x6TQATpQBholQzxaVelm7JYW5GFh9tzcXr1p+bnhRn9eeMgLAz9yhPpn7NBxvNqdeYMVEKECZK/tnmU38uz60/N3VYD6YNt/pzRv2wOrOAPy3aYk69RoqJUoAwUTrJ0aJSFm7Yy9z0LNbsPkhIkFN/bnpSHKP7Wf05o/5RVTybc3jcdeoNT3CcesMSzKnX0JgoBYiWLkqqyprdTv25/244WX9u+vA4qz9nNBrKnXrPLNlG7pEiJl3YjQcnmVOvITFRChAtVZQOHHXqz81Nd+rPRYQG8x2rP2c0csqderM/2kGhOfUaFBOlANGSRMlf/bmh8R2ZnhTH1YndaWsuJ6OJUO7U+8+q3YSZU69BMFEKEC1BlHbnHa+oP/ft4UKi2oRy/VDHyt23q9WfM5ouvk696LZh/CzZnHr1hYlSgGiuolRYUsbijd8yNz2LlTvzrP6c0axZnVnA4x9sIX1XAb1dp16KOfUCiolSgGhuorRxzyHeysjivbV7OFxYSlxka6YPj7P6c0azx5x69YuJUoBoDqJ06HgJ76936s9t2nuYULf+3PThcVzc2+rPGS2L0jIvb2Vk8/SSrebUCyAmSgGiqYqS16t8vjOPuRlZLN74LUVu/bnpSXFMtvpzhnGKU2/GiDjuG9/PnHp1RKMQJRGZBDwLBAMvqOrjVd4PA14FhgF5wHRV3eW+lwj8E2gPeIEkVS0UkcVADBACfAz8WFXLRGQucL576I7AQVUdLCK3AA/4nDYRGKqq60RkuXusE+57Kaq6v6ZramqiVF5/bt7qbHbnH6ddeAjXDY5lepLVnzMMf1R16t01ujd3jeptTr1zpMFFSUSCga1AMpANpAMzVHWzT597gERVvVtEbgKmqOp0EQkB1gC3qup6EYnCEZkyEWmvqofFWZGcD8xT1TlVzv2/wCFVfaxK+yDgfVXt7b5eDvxCVWutMk1BlIpLvSz7Koe56VmscOvPXdLbqT83aaDVnzOM2mBOvbqltqIUSOkfAWxX1Z1uQHOAycBmnz6TgUfd5/OB512xSQE2qOp6AFXNKx+gqod9Yg8FKqmqO34aMM5PTDOA/5zTVTVitu936s+9s+Zk/bl7rujDjcN7kBDVpqHDM4wmRe/ObfnHd4dVOPV+/e5GXvzkG3PqBZhAilIskOXzOhsYWV0fVS0VkUNAFNAPUBFJBToDc1T1z+WD3PYRwAc4YubLKCBHVbf5iWk6jhD68rKIlAFvA3/QJrbIdqyolP9a/TnDCBjDEjrx1g8vwePufvvD11YzLKETv7rqAoYlRDZ0eM2OQIqSv2/Dql/41fUJAS4HkoDjwFI39VsKoKoTRSQceAMnI/L4jPebDYnISOC4qm70ab5FVfeISDscUboVZ42r6tiZwEyA+Ph4PyHXL079uYO8lZ7Ffzfs5VhxGed1bsOvrrqAKUN62MKsYdQxIkLKhd0Yd0GXCqfeDf9YaU69ABBIUcoG4nxe9wD2VtMn211H6gDku+0rVPUAgIgsAoYCS8sHuqaHBTiZj8ftFwJcj2OcqMpNVBErVd3j/jwiIm/iZF+niJKqzgZmg7OmVItrDwgHjhbx7po9zM3IYvv+o0SEBnNNYgzTk+IYGt/JphMMI8CEBAdx88h4rhvS3dn9dsUOPFtyzKlXhwRSlNKBviLSC9iDIwo3V+mzALgdWAlMBZapavm03YMiEgEUA2OAp0WkLdBOVfe5AnQVjgOvnAnAV6qa7XsSEQkCbgRG+7SFAB1V9YCItAKuAZbU0bXXGWVe5aOtTv25JVtyKurPPXHDIKs/ZxgNRERoCD8d35ebR8Y7u99+sZt31uxhpjn1zpmAfXLuGtG9QCqOJfwlVd0kIo8BGaq6AHgReE1EtuNkSDe5YwtE5CkcYVNgkaouFJGuwALXSh4MLANm+Zz2lGzIZTSQXW66cAkDUl1BCsYRpH/V1fWfK7vzjjNvdRbzMpz6c5FtQrnj0p5MT7L6c4bRWIhuG8Zjkwdyx6U9eTL1a55Zso3XP9/t7H6bFEcrc+qdMXbz7BkSSEt4YUkZqZuc+nOf7XDqz43u15npw+MY39/qzxlGY8dq6lVPg9+n1FwJhCj5qz83bVgcU4db/TnDaGqoKku27OfxD7awI/eYOfVcTJQCRF2J0qETJSxY55gWNu5x6s9NurAb05PiuMTqzxlGk8dfTb0HJp3PeS3UqWeiFCDORZS8XuXzb/J4Kz2LD9z6cwNi3Ppzg7vTMSK0jqM1DKOhOV5cWuHUa8k19UyUAsTZiJLXq/xjxQ7mpmdZ/TnDaKEcOFpU4dQLDQlqcU49E6UAcbaZ0rRZKwkOEqs/ZxgtnG8OHOPJ1K9Y9KVTU6+lOPVMlALE2YpSYUmZCZFhGBWs2V3AnxaddOo9OPECJl7YfJ16tRWl5i3NjQgTJMMwfBka79TU+9dtwxHg7tdXM3XWSlZn5jd0aA2KiZJhGEYDISIkD+hK6v2j+dP1g9idf5wb/rGSH76WwY7cow0dXoNg03dnSFPYT8kwjKZJVafeTUlx3DehL13ahTd0aOeMrSkFCBMlwzACTXN06pkoBQgTJcMw6ovm5NQzo4NhGEYTp1d0G/5+yzDeuedSeke34TfvbWTi0x+xeOO3NNeEwkTJMAyjkTM0vhNzf3gx/7ptOEFB0qydeiZKhmEYTYByp97i+0Y1a6eerSmdIbamZBhGY+B4cSkvfvwNs5qIU8+MDgHCRMkwjMbEgaNFPLd0G2+4Tr27RvVm5ujG59QzUQoQJkqGYTRGvjlwjL+kfs3CL/c1Sqeeue8MwzBaEL2i2/C3W4Y2eadeQEVJRCaJyNcisl1EHvLzfpiIzHXf/0JEevq8lygiK0Vkk4h8KSLhbvtiEVnvts8SkWC3fa6IrHMfu0RkndveU0RO+Lw3y+ccw9xjbxeRv0pzrYRoGEaLodyp90IVp17Grqbh1AvYpKMrFn8DkoFsIF1EFqjqZp9u3wcKVLWPiNwEPAFMF5EQ4HXgVlVdLyJRQIk7ZpqqHnYFZD5wIzBHVaf7nPt/gUM+59mhqoP9hPkPYCbwObAImAR8cM4Xb1RLSUkJ2dnZFBYWNnQohlErwsPD6dGjB61atWroUGqNiDBhQFeuOL8z81Zn87RnK1NnrWTihV15cNIFjXr320CuhI0AtqvqTgARmQNMBnxFaTLwqPt8PvC8KzYpwAZVXQ+gqnnlA1T1sE/soUClvNQdPw0YV1NwIhIDtFfVle7rV4HrMFEKKNnZ2bRr146ePXs22xL9RvNBVcnLyyM7O5tevXo1dDhnTEhwEDNGxDN5cHde/Pgb/vnRTlKe/qhRO/UCOX0XC2T5vM522/z2UdVSnOwmCugHqIikisgaEXnQd5CIpAL7gSM4YubLKCBHVbf5tPUSkbUiskJERvmcO/s08Rl1TGFhIVFRUSZIRpNARIiKimrymX1EaAg/Gd+X5Q9cwXdHxjM3PYsrnlzO056tHCsqbejwKhFIUfL3rVN1ta26PiHA5cAt7s8pIjK+ooPqRCAGCOPUjGgG8B+f1/uAeFUdAvwceFNE2tcyPidIkZkikiEiGbm5uf66GGeACZLRlGhOv6/RbcP4/eSBeH4+hrHnd+HZpdsY8+RyXvs8k5Iyb0OHBwRWlLKBOJ/XPYC91fVx15E6APlu+wpVPaCqx3HWe4b6DlTVQmABzhQgPse4Hpjr06+ofPpPVVcDO3AysWw3ppriKz/GbFUdrqrDO3fuXKuLNwzDaKyUO/XedZ16jzQip95pRUlEzhORMPf5FSLyUxHpWItjpwN9RaSXiIQCN+GIiC8LgNvd51OBZep8IqlAoohEuEIzBtgsIm3dtaByAboK+MrneBOAr1S1YlpORDr7OPR6A32Bnaq6DzgiIhe761C3Ae/X4rqMJs7Bgwf5+9//flZjn3nmGY4fP15nsbzyyivce++9dXa82rJ8+XKuueaaMx53pvHu2rWLN99884zPU1vy8/NJTk6mb9++JCcnU1BQcEqfDz/8kMGDB1c8wsPDee+99wC444476NWrV8V769atC1isjZEhjdCpV5tM6W2gTET6AC8CvYDT/pa5a0T34gjMFuAtVd0kIo+JyLVutxeBKBHZjjO19pA7tgB4CkfY1gFrVHUh0AZYICIbgPU460qzfE57E5Wn7gBGAxtEZD3O+tPdqlr+if8IeAHYjpNBmcmhBdCYRKm5U5MolZae+1rG448/zvjx49m2bRvjx4/n8ccfP6XP2LFjWbduHevWrWPZsmVERESQkpJS8f6TTz5Z8f7gwf5Mus2bcqfe4vtG8fj1g8jKP87UWSuZ+WoG2/fXf0292rjvvKpaKiJTgGdU9TkRWVubg6vqIpypN9+23/o8L8SxdPsb+zqOLdy3LQdIquF8d/hpextHWP31zwAGVnsBRkD5/f9tYvPew6fveAYM6N6e333nwhr7PPTQQ+zYsYPBgweTnJxMly5deOuttygqKmLKlCn8/ve/59ixY0ybNo3s7GzKysp45JFHyMnJYe/evYwdO5bo6Gg+/PBDv8dfvHgxv/rVrygrKyM6OpqlS5eSn5/P9773PXbu3ElERASzZ88mMTGx0rh58+bx+9//nuDgYDp06MBHH33Erl27uPXWWzl27BgAzz//PJdeeinLly/nd7/7HV27dmXdunVcf/31DBo0iGeffZYTJ07w3nvvcd5553HHHXcQHh7Opk2byMnJ4amnnjolQzp27Bg/+clP+PLLLyktLeXRRx9l8uTJVEdWVhaTJk3im2++4eabb+Z3v/sdjzzyCNHR0dx3330A/PrXv6Zr1668+eabbNmyhcGDB3P77bfTqVMnFi5cSGFhIceOHWPZsmU8+eSTp3z+AK+//jp//etfKS4uZuTIkfz9738nODi4Uizvv/8+y5cvB+D222/niiuu4Iknnqg29vnz53PllVcSERFRbZ+WSkhwEDeNiOdaH6fexGfq36lXG1EqEZEZONNs33Hbmo5h3zCq8Pjjj7Nx40bWrVtHWloa8+fPZ9WqVagq1157LR999BG5ubl0796dhQsXAnDo0CE6dOjAU089xYcffkh0dLTfY+fm5nLXXXfx0Ucf0atXL/LznaT8d7/7HUOGDOG9995j2bJl3HbbbadMFT322GOkpqYSGxvLwYMHAejSpQsej4fw8HC2bdvGjBkzKC9ztX79erZs2UJkZCS9e/fmBz/4AatWreLZZ5/lueee45lnngGcbGXFihXs2LGDsWPHsn379krn/eMf/8i4ceN46aWXOHjwICNGjGDChAm0adPG7zWuWrWKjRs3EhERQVJSEldffTXf//73uf7667nvvvvwer3MmTOHVatWkZiYyF/+8hf++9//As7038qVK9mwYQORkZGkpaWxbdu2Uz7/zp07M3fuXD799FNatWrFPffcwxtvvMFtt93GD37wA+6++26GDx9OTk4OMTExAMTExLB///4a/+3nzJnDz3/+80ptv/71r3nssccqMq2wsLAaj9HcKXfqzRgZX1FT7921e7hrVG9+OKY3EaGBralXm6PfCdwN/FFVvxGRXlTJYAzjbDhdRlMfpKWlkZaWxpAhQwA4evQo27ZtY9SoUfziF7/gl7/8Jddccw2jRo06zZEcPv/8c0aPHl1xT0tkZCQAn3zyCW+/7STs48aNIy8vj0OHDlUae9lll3HHHXcwbdo0rr/+esC52fjee+9l3bp1BAcHs3Xr1or+SUlJFV/I5513XsWU1KBBgyplcdOmTSMoKIi+ffvSu3dvvvrKdxnW+QwWLFjAX/7yF8Cx7e/evZv+/fv7vcbk5GSioqIAuP766/nkk0+4//77iYqKYu3ateTk5DBkyJCKPv7Gl38u1X3+GzZsYPXq1SQlORMjJ06coEuXLgC88MILfo97Ovbt28eXX37JxIkTK9r+9Kc/0a1bN4qLi5k5cyZPPPEEv/3tb2s4Ssuh3Kl352W9eDL1a/69chffu6yXc3doADmtKLkVGH4KICKdgHaqeurErWE0QVSVhx9+mB/+8IenvLd69WoWLVrEww8/TEpKSq2+rFTVr4XYn6Opar9Zs2bxxRdfsHDhwopF9+eee46uXbuyfv16vF4v4eEnp1B8/6IPCgqqeB0UFFRpvabqeaq+VlXefvttzj///NNeX03H+8EPfsArr7zCt99+y/e+971qx/tmYNV9/s899xy33347f/rTn2qMpWvXruzbt4+YmBj27dtXIVz+eOutt5gyZUqlygzloh4WFsadd95ZIczGSXq6Tr38Y8V0iAj8JFlt3HfLRaS9iETimAteFpGnAh6ZYQSIdu3aceTIEQAmTpzISy+9xNHxEvm+AAAgAElEQVSjzoLunj172L9/P3v37iUiIoLvfve7/OIXv2DNmjWnjPXHJZdcwooVK/jmm28AKqbvRo8ezRtvvAE4zrfo6Gjat29faeyOHTsYOXIkjz32GNHR0WRlZXHo0CFiYmIICgritddeo6ys7Iyvd968eXi9Xnbs2MHOnTtPEZ+JEyfy3HPPVQjn2rU1Lxl7PB7y8/Mr1q4uu+wyAKZMmcLixYtJT0+vyEZO93lV9/mPHz+e+fPnV0zH5efnk5mZecr4a6+9ln//+98A/Pvf/65xLew///kPM2bMqNS2b98+wBHH9957j4EDbYm5OiLbBDhFcqnN9F0Ht9bcD4CXVfV3rvvNMJokUVFRXHbZZQwcOJArr7ySm2++mUsuuQSAtm3b8vrrr7N9+3YeeOABgoKCaNWqFf/4xz8AmDlzJldeeSUxMTF+jQ6dO3dm9uzZXH/99Xi93oo1oUcffZQ777yTxMREIiIiKr5IfXnggQfYtm0bqsr48eO56KKLuOeee7jhhhuYN28eY8eOrXadpybOP/98xowZQ05ODrNmzaqUbQE88sgj3H///SQmJqKq9OzZs2INyB+XX345t956K9u3b+fmm29m+HBnN4LQ0FDGjh1Lx44dKwwJiYmJhISEcNFFF3HHHXfQqVOnSsdKSUlhy5Ytp3z+AwYM4A9/+AMpKSl4vV5atWrF3/72NxISEiqtKT300ENMmzaNF198kfj4eObNmwdARkYGs2bNqpjq27VrF1lZWYwZM6bS+W+55RZyc3NRVQYPHsysWbMwGpbT7qckIl/i1KL7N/BrVU0XkQ2qmljjwGaK7ad0bmzZsqXatQqj7rnjjju45pprmDp1asDP5fV6GTp0KPPmzaNv374BP199Yr+3505d7qf0GM69RjtcQeoNbDvNGMMwWhCbN2+mT58+jB8/vtkJklG/1MboMA+Y5/N6J3BDIIMyjKbAyJEjKSoqqtT22muvMWjQoAaK6FReeeWVsxqXmprKL3/5y0ptvXr14t133/Xbf8CAAezcufOszmUYvpxWlESkB/AccBlOwdJPgPt8S/kYRkvkiy++aOgQAsbEiRMrWacNo76ozfTdyzg16rrjbO3wf26bYRiGYdQptRGlzqr6sqqWuo9XACuVbRiGYdQ5tRGlAyLyXREJdh/fBfJOO8owDMMwzpDaiNL3cLYX/xZnw7ypOKWHDMMwDKNOOa0oqepuVb1WVTurahdVvQ5nIz3DaJI0pq0rbD+lc6M2+ykBPPjgg1x44YX079+fn/70pw2+kZ1RPWe78+zPT9/FMBonjUmUmjuNYT+lzz77jE8//ZQNGzawceNG0tPTWbFixTmf2wgMZ1uDvPlsWm80HB88BN9+WbfH7DYIrqy5XrDtp9Sy9lMSEQoLCykuLkZVKSkpoWvXrjX+jhgNx9mKkuW+RpPF9lNqWfspXXLJJYwdO5aYmBhUlXvvvddKBjViqhUlETmCf/ERoHXAIjJaDqfJaOoD20+p+e+ntH37drZs2UJ2dnbF+T/66CNGjx59Rscx6odqRUlV253rwUVkEvAsEAy8UHUfJhEJA14FhuHYzKer6i73vUTgn0B7wAskqWqhiCwGYtzYPwZ+rKplIjIXKK/J3xE4qKqDRSQZeBxna6pi4AFVXeaeY7l7rBPuuBRVrXnrSqNZYfspNf/9lN59910uvvhi2rZtC8CVV15Z8ceD0fg4W6PDaRGRYOBvwJXAAGCGiAyo0u37QIGq9gGeBp5wx4bg7G57t6peCFwBlLhjpqnqRcBAnJt4bwRQ1emqOlhVBwNvA++4/Q8A31HVQThbur9WJYZbyseZILUMbD+llrWfUnx8PCtWrKC0tJSSkhJWrFhh03eNmEButj4C2O4WcEVE5gCTgc0+fSYDj7rP5wPPi/NnVwqwQVXXA6hqxc26qnrYJ/ZQqkwxuuOnAePc/r7/wzYB4SISpqqVK2kaLQbbT6ll7ac0depUli1bxqBBgxARJk2axHe+850z/hyNekJVA/LAucn2BZ/XtwLPV+mzEejh83oHEA3cj5PRpAJrgAerjEsFCoA3geAq740GMmqIaYnP6+XAl8A64BHc/aVqegwbNkyNs2fz5s0NHUKL4vbbb9d58+bVy7nKysr0oosu0q1bt9bL+eoT+709d6r7Xq76CNj0Hf5t41Un1qvrEwJcDtzi/pwiIuMrOqhOxFkLCsPNiHyYAfznlGBELsSZHvSdvL5FnWm9Ue7jVr8XIjJTRDJEJCM3N9dfF8No0dh+SkZdUZutK/y58A4BGcD/p+70nB+ygTif1z2AvdX0yXbXkToA+W77ClU94MawCBgKLC0fqI7pYQHOFKDH7ReCU21iWJVr6AG8C9ymqjt8jrHH/XlERN7EmXJ8teqFqOpsYDY4O89Wc71GC8P2UzqJ7adk1BW1WVN6CkdM3sTJbG4CugFfAy/hmBD8kQ70FZFewB533M1V+izAMR+sxJlaW6aqKiKpwIMiEoHjmBsDPC0ibYF2qrrPFaCrcBx45UwAvlKfvZ5EpCOwEHhYVT/1aQ8BOqrqARFpBVwDLKnF52EYgO2nZBiBoDbTd5NU9Z+qekRVD7tZw1WqOhfoVN0gVS0F7sVZ/9kCvKWqm0TkMRG51u32IhAlIttxShc95I4twBHDdJz1njWquhBoAywQkQ3AemA/MMvntDdx6tTdvUAf4BERWec+uuBM/aW6x1qHI5z/qsXnYRiGYQSI2mRKXhGZhuOOAyejKafGqSxVXQQsqtL2W5/nhbiWbj9jX8exhfu25QBJNZzvDj9tfwD+UM2QYdW0G4ZhGA1AbTKlW3AMAPvdx63Ad0WkNU4WYhiGYRh1wmkzJdfIUJ2p/5O6DccwDMNoyZw2UxKRHiLyrojsF5EcEXnbdbMZRpOkMW1dYfspnRvnup/SFVdcwfnnn8/gwYMZPHiw34KuRv1Sm+m7l3Fcct2BWOD/3DbDaJI0JlFq7jSF/ZTeeOMN1q1bx7p16/zWzjPql9oYHTqrqq8IvSIi9wcqIKPl8MSqJ/gq/6vTdzwDLoi8gF+O+GWNfWw/JdtPyfZTarzURpQOiMh3OWm1noFT0dswmiS2n5Ltp+RbkPXOO+8kODiYG264gd/85jd+q7wb9UdtROl7wPM4VbwV+Ay4M5BBGS2D02U09YHtp9Sy91N64403iI2N5ciRI9xwww289tpr3HbbbWd0fKNuqY37bjdwrW+bO333TKCCMoz6Qm0/pRa9n1JsbCzgbLFx8803s2rVKhOlBuZsC7L+vE6jMIx6xPZTsv2U+vfvT2lpKQcOHACcjPS///0vAwcOrPHajcBztvsp2aSr0WSx/ZRsP6XvfOc7HDt2jIkTJ1JSUkJZWRkTJkzgrrvuOuPP16hbxN+0wmkHiexW1fgAxNPoGT58uJYvNBtnzpYtW2zXz3rkjjvu4JprrmHq1Kmn73yOeL1ehg4dyrx585rd9hX2e3vuiMhqVR1+un7VTt+JyBEROezncQTnniXDMAzA9lMy6o5qp+9UtV19BmIYTQ3bT+kktp+SUVec7ZqSYZw11TnUmhq2n1LL4GyWOIyzJ5DboRs+PL/2eV7e+DJZR7IaOpQGJTw8nLy8PPuPbjQJVJW8vLxTzCFG4LBMqR5QVdbsX0P6t+k8tfop+kf2J6VnCskJySS0T2jo8OqVHj16kJ2dTW5ubkOHYhi1Ijw8nB49rAZ1fXFW7ruWzLm47/Yc3cOSzCWk7Upjw4ENAPTr1I+UhBSSeybTu0PvugzVMAyj0VBb952J0hlSV5bwfUf3sWT3EjyZHtbud25W7NOxD8kJySQnJNOnY59mse5iGIYBjUSURGQS8CwQDLygqo9XeT8MeBVnW/I8YLqq7nLfSwT+CbQHvECSqhaKyGIgBmfq8WPgx6paJiJzgfJb1TsCB1V1sHush4HvA2XAT1U1tTbx+SMQ9ynlHMth6e6leDI9rM5ZjaL06tCL5IRkUhJS6NepnwmUYRhNmgYXJREJBrYCyUA2kA7MUNXNPn3uARJV9W4RuQmYoqrTRSQEWAPcqqrrRSQKR2TKRKS9qh4W51t6PjBPVedUOff/AodU9TERGYBT4XwEzv1VS4B+btca4/NHoG+ePXDiAEszHYFKz0nHq17i28U7GVTPZAZEDjCBMgyjyVFbUQqk0WEEsN3dTh0RmQNMBny/9CcDj7rP5wPPu2KTAmxQ1fUAqlqxVYaqHvaJPRSncnkF7vhpwDifc8xR1SLgGxHZ7sZGLeKrd6JbRzP9gulMv2A6+YX5LNu9jLRdabyy6RVe3PgisW1jnTWohGQGRg80gTIMo1kRSFGKBXz9z9nAyOr6qGqpiBwConAyGRWRVKAzjqj8uXyQ2z4C+ABHzHwZBeSo6jafc3xeJY5Y9/np4mtQIsMjmdpvKlP7TeVg4UE+zPqQtMw0XtvyGi9vepmYNjFMSJhASkIKiZ0TCRJz+BuG0bQJpCj5+xO+6lxhdX1CgMuBJOA4sNRN/ZYCqOpEEQkH3sDJiDw+42dwckPCms7h7xvc71ymiMwEZoJTcbgh6BjekSl9pzCl7xQOFR1iRfYKPLs8zPlqDq9tfo0uEV0qTBKDOw8mOCj49Ac1DMNoZARSlLKBOJ/XPYC91fTJdteROgD5bvsKVT0AICKLgKHA0vKBrulhAc6Um8ftFwJcj2OcqE0cp4uv/FyzgdngrCnVdNH1QYewDlx73rVce961HC0+yvLs5Xh2eZj39Tze2PIG0a2jGR8/npSEFIZ2HUpIkN2OZhhG0yCQ31bpQF8R6QXsAW4Cbq7SZwFwO7ASmAosU9XyabsHRSQCKAbGAE+LSFugnarucwXoKhwHXjkTgK9UNbvKOd4UkadwjA59gVU4GdTp4mv0tA1tyzW9r+Ga3tdwrOQYH2d/TFpmGu9vf5+5X88lMjyScfHjSElIIalbkgmUYRiNmoB9Q7lrRPcCqTiW65dUdZOIPAZkqOoC4EXgNdd8kI8jDKhqgSsi6ThTaotUdaGIdAUWuFbyYGAZMMvntDdReeoO95xv4RgYSnEt5AD+4gvIh1FPtGnVhkm9JjGp1ySOlxznkz2f4Mn0sHDnQuZvnU/HsI6Mix9HckIyI7uNpFVwq4YO2TAMoxJ28+wZ0hT3UyosLeTTvZ/iyfSwPGs5x0qO0S60HWPjxjKx50QujrmY0ODQhg7TMIxmTIPfp9RcaYqi5EtRWREr967Ek+nhw90fcqTkCG1bteWKuCtITkjmstjLCAsOa+gwDcNoZpgoBYimLkq+lJSV8Pm+z/FkeliWtYxDRYeICIlgTI8xpPRM4bLYy2gd0rqhwzQMoxlgohQgmpMo+VLiLSF9XzppmWks272MgqICWoe0ZlTsKJJ7JjM6djQRrSIaOkzDMJooJkoBormKki+l3lJW56zGk+lhSeYS8grzCAsO4/LYy0lOSGZMjzG0DW3b0GEahtGEMFEKEC1BlHwp85axdv9a0jLTWJK5hNwTuYQGhXJp7KWkJKQwJm4M7UPbN3SYhmE0ckyUAkRLEyVfvOplfe560nal4cn0kHM8h5CgEC6JuYTkhGTGxY+jQ1iHhg7TMIxGiIlSgGjJouSLV71sPLCxQqD2HttLiIQwMmZkhUB1Cu/U0GEahtFIMFEKECZKp6KqbM7bTFpmGmm70sg+mk2wBDO823BSElIYFz+O6NbRDR2mYRgNiIlSgDBRqhlV5euCr0nblUZaZhqZhzMJkiCGdR1GckIyE+In0Dmic0OHaRhGPWOiFCBMlGqPqrLt4DY8mR48uzzsOLQDQRjSZYgjUAkT6NamW0OHaRhGPWCiFCBMlM6eHQd34Mn0kJaZxrYCZ7urizpfVLHlRve23Rs4QsMwAoWJUoAwUaobvjn0DUsyl+DJ9LAlfwsAA6MGktzTEai4dnGnOYJhGE0JE6UAYaJU92QdzsKz20ParjQ25TmF2vtH9ielp7Pte0L7hAaO0DCMc8VEKUCYKAWWPUf3sCRzCWmZaWzI3QBAv079SE5IJiUhhd4dezdwhIZhnA0mSgHCRKn++PbYtxVTfGv3r0VRzutwXkUG1adjH0T87XZvGEZjw0QpQJgoNQz7j++vEKjVOatRlJ7tezoZVM8Uzu90vgmUYTRiTJQChIlSw3PgxAGW7V5GWmYa6d+m41Uv8e3iHRdfz2QGRA4wgTKMRoaJUoAwUWpc5Bfms2z3MjyZHr7Y9wVlWkZs29gKm/mg6EEmUIZxrhzeB3tWQ/9rzvoQjUKURGQS8CwQDLygqo9XeT8MeBUYBuQB01V1l/teIvBPoD3gBZJUtVBEFgMxQAjwMfBjVS1zx/wEuBcoBRaq6oMicgvwgM9pE4GhqrpORJa7xzrhvpeiqvtruiYTpcbLwcKDfJj1IZ5MDyv3raTUW0q3Nt0qTBKJnRMJkqCGDtMwGj9lJZC1CrZ7YNsSyPkSEHhgO7Q5u5JhDS5KIhIMbAWSgWwgHZihqpt9+twDJKrq3SJyEzBFVaeLSAiwBrhVVdeLSBRwUFXLRKS9qh4W58/f+cA8VZ0jImOBXwNXq2qRiHSpKjAiMgh4X1V7u6+XA79Q1VqrjIlS0+Bw8WGWZy3Hs8vDp3s/pcRbQpfWXZiQMIHkhGSGdBlCcFBwQ4dpGI2Hw/tcEfLAzuVQdBiCQiDuYug7AfqmQJcBcJYzD7UVpZCzOnrtGAFsV9WdbkBzgMnAZp8+k4FH3efzgeddsUkBNqjqegBVzSsfoKqHfWIPBcpV9UfA46pa5Pbzl/HMAP5zzldmNHrah7bn2vOu5drzruVo8VFWZK/Ak+nh7W1v8+ZXbxLdOprx8eNJSUhhaNehhAQF8r+CYTRC/GZDQLvucOF10CcZel8B4fW7X1og/yfGAlk+r7OBkdX1UdVSETkERAH9ABWRVKAzMEdV/1w+yG0fAXyAI2a4Y0aJyB+BQpwMKL3K+abjCKEvL4tIGfA28Ae1RbZmR9vQtlzd+2qu7n01x0qO8XH2x6RlpvH+9veZ+/VcIsMjGRc/juSEZJK6JdEqqFVDh2wYgeHwXti+5NRsKP4SmPB76Jt8TtlQXRBIUfJ3VVW/8KvrEwJcDiQBx4Glbuq3FEBVJ4pIOPAGMA7wuGM6ARe7494Skd7lIiMiI4HjqrrR51y3qOoeEWmHI0q34qxxVQ5SZCYwEyA+Pr421240Utq0asOkXpOY1GsSx0uO8+neT/Hs8rBw50Lmb51Px7COFQI1sttIWgWbQBlNmLISyPrCEaHtSyDH/fpr4GyoJgIpStmAbwGzHsDeavpku+tIHYB8t32Fqh4AEJFFwFBgaflA1/SwACfz8bhj3nFFaJWIeIFoINcdchNVpu5UdY/784iIvImTfZ0iSqo6G5gNzprSmX0MRmMlolVEhUuvsLSQz/Z+RlpmGqm7Unln2zu0C23H2LixpCSkcEn3SwgNDm3okA3j9DSBbKgmAilK6UBfEekF7MERhZur9FkA3A6sBKYCy1S1fNruQRGJAIqBMcDTItIWaKeq+1wRuwrHgQfwHk7WtFxE+uGsN5WLWhBwIzC6/MTu+I6qekBEWgHXAEvq+kMwmgbhIeGMix/HuPhxFJcVs3LvStIy0/gw60MW7FhA21ZtGRM3hpSEFC7tfinhIeENHbJhOFSXDbWPhQunOCLUa8wZZUOl3lIOFh0kvzCf/MJ8CgoLKCgsYMYFMwJ+i0XARMldI7oXSMWxhL+kqptE5DEgQ1UXAC8Cr4nIdpwM6SZ3bIGIPIUjbAosUtWFItIVWOBayYOBZcAs95QvAS+JyEYcIbvdZ31oNJBdbrpwCQNSXUEKxhGkfwXm0zCaEqHBoYyJG8OYuDGUlJXw+b7P8WR6WJa1jIU7FxIREsGYHmNI7pnM5bGX0zqkdUOHbLQ0Du91RcgDO1ecNhsq85ZxyBWXcqHJ9/O6vO1Q0SH0lNUWuK7PdUS0igjopdnNs2eIWcJbLiXeEtK/TceT6WFp5lIKigpoHdKaUbGjSO6ZzOjY0QH/D2u0UKpkQ5qzkcNBQeS3jyE/bjgF3fqT36E7eWXHK4TGV3AOFh3Eq16/h+4Y1pFO4Z2IDI+s9Kja1im8E53COp11ptTg9yk1V0yUDHCmN9bkrCEtM40lmUvIK8wjLDiMy2MvJzkhmTE9xtA2tG1Dh2k0EVSVoyVHK4QkrzCPgoJvyN+7moK8r8g7spcCKSM/OIT8VuEcFC+lfjIZgHah7YgKj6oQlZoEp2NYx3q7HcJEKUCYKBlVKfOWsXb/WjyZHpZkLmH/if20CmrFZd0vI6VnCmPixtA+tPG4m4zAo6qcKD1xyrRYXmHeKZlMeVuJt8Tvsdp4lciQCDq1jiayQwKRbbpUn82EdWq0jlETpQBx1qK0fQmEd4JuAyEkrO4DMxoFXvWyIXcDqbtS8WR6yDmeQ0hQCJfEXEJyQjLj4sfRIaxDQ4dpnAWFpYXVikteYV4loSkoLKCwrNDvcVqHtK4QkMiQCCKLjtPpSA5R+bvpVHycSC906nIhUT2voFO/qwmLuajROuXOBBOlAHHWovTsRVCwC4JaQdcLIXYodB/q/Iw+H4KtokBzw6teNh7YiCfTgyfTw56jewiREEbEjCAlIYVx8ePoFN6pocNssRSXFde42F9VgI6XHvd7nNCgUCJbn8xcosKjHMFp7QhPVOuok2syrdoRsXf9ySoK+52dlmkfC33cUj69x0BYu3r8JOoHE6UAcdaidDAL9q6BPWucn3vXOY4ZgFYR0C2xslBF9m4Wfx0ZDqrK5vzNeHZ5SMtMI+tIFsESzPBuwysEKrr12RW6NBxKvCUcLDxYo7vM9/XRkqN+jxMSFEJkWGSFqJwiLm5beZ+IkIiaF/8P7fGpKbcCio84f5zGX+y45PokQ5f+zf7/u4lSgKizNSWvF/J3nBSpPWvg2w1Q6qb84R2g+5CTItV9KLTv3ux/cVsCqsrXBV+TtisNT6aHXYd3IQjDug4jOSGZCQkT6BLRpaHDbHDKvGUV98rUxsp8uPiw3+MESzAdwzqeFJLwyoJTVYDatWp3bvfilJXA7s/9ZEM9nMKmfZKbbTZUEyZKASKgRoeyEti/BfauPSlU+zeDt9R5v23XyiIVOxQiIgMTi1EvqCrbD27Hk+khbVcaOw7tQBCGdBlSIVDd2nRr6DDrBK96OVx0uFb3yZTbmP3dKyOIIzKncZaVC077sPaB37Kkumwo4RJHhPomQ+cLWvQflSZKAaLe3XclJ+DbjZWn/g5so6KMYMeEyiIVc1GL+wusObHj4I6KNaitBVsBSOycSEpCCskJyXRv272BIzyJqnKk5Aj5J/IpKCog/0Q++UX5zk9fwXHbDhYdpMzZ+uwU2oe2P+19MuVtHcM6Nvy2I6XFkPX5ySoK+93ND1p4NlQTJkoBolFYwgsPw751PlN/a+HQbvdNgc7n+2RUQ6DrQGhlZXGaGrsO7WLJ7iWk7UpjS/4WAAZGDSS5p1OvL65d3GmOcGaoKsdLj58qLkUF5J3IqxAeXwEqLc/iq9CuVbta3ScTGR5Jx/COTaMyu2VD54SJUoBoFKLkj6O5laf99q6BY24t2qBW0HVA5am/zheY468JkXU4C89uD55dHjbmObXN+kf2J6VnChPiJ9CzQ0+/4yrulTnhX1zyCvMqCU2xt9jvcSJCIk46y3wExd9d/5Hhkc2jeO3psqG+KdBrtGVDtcREKUA0WlGqiiocyjbHXzNkz9E9LMlcgifTw/rc9QD069SPCyIvcIwB5SJTmM+J0hN+jxEWHHaKmFStAhAVftLK3GIK0Fo2FDBMlAJEkxElf3i9kL+zslDt2wDlX1zm+GtyfHvs2wqB2nN0zykZS1VxKX+/dUjrgFd7bhJUlw11iHPvG0q2bKiOMFEKEE1alPxRVgq5Wypb02ty/HUfAm2iGjZmwzgXDmWfFKGdy6H4aJVsKMVZlzXRrlNqK0q2qNDSCQ6BboOcx7DbnTZ/jr+tiznV8edmVd0H21+SRuOlIhtKc+4bynVMI3SIg0E3WjbUyDBRMk6lVWuIS3Ie5VR1/GWvhk3vum8KRPervD5ljj+jIak2G7oUhtziZESWDTVKTJSM2hHe3vlrstfok23HDviYKNbC9qWw3t1x3hx/Rn1SWgy7V56somDZUJPF1pTOkGa3plSXqMLhPZXXp/aug6JDzvvm+DPqkoNZJ0XomxWVs6HymnKWDTUabE3JqH9EoEMP5zHgWqfNn+Mv42Uo/bvz/imOvyFOxWT7IjGqUm02FA+J0xwR6jUawmxzxaZMQEVJRCYBzwLBwAuq+niV98OAV4FhQB4wXVV3ue8lAv8E2gNeIElVC0VkMRDjxv4x8GNVp3aJiPwEuBcoBRaq6oMi0hPYAnztnvZzVb3b7T8MeAVoDSwC7lNLHeuWoCCI7uM8Eqc5bf4cf5/9tYrjr4o13Rx/LZOasqEhtzhOueh+9kdMMyJgoiQiwcDfgGQgG0gXkQWqutmn2/eBAlXtIyI3AU8A00UkBHgduFVV14tIFFC+LeM0VT0szk0W84EbgTkiMhaYDCSqapGI+JZZ3qGqg/2E+Q9gJvA5jihNAj6om0/AqJZaO/5SOen4i69iTTfHX7OkUjbkgdyvnHbLhloMgcyURgDbVXUngIjMwRENX1GaDDzqPp8PPO+KTQqwQVXXA6hqXvkAVS2vTx8ChFLxrcWPgMdVtcjtt7+m4EQkBmivqivd168C12Gi1DBU6/hbX1moNr/nvmmOv2aDv2woONTNhm511ocsG2oxBFKUYoEsn9fZwMjq+qhqqYgcAqKAfoCKSCrQGZijqn8uH+S2j8ARkPlucz9glIj8ESgEfqGq6e57vURkLXAY+I2qfuyeO7tKfLHndslGnRLeHnqNch7lHDvgOP3KRaqS4y/E2dXXHH+Nm9Ji2P3ZScu2ZUOGD4H83+rvz8AcUiYAAApySURBVJqq6zXV9QkBLgeSgOPAUte5sRRAVSeKSDjwBjAO8LhjOgEXu+PeEpHewD4gXlXz3DWk90TkwlrG5wQpMhNnmo/4+Pjqr9gIPG2inb+c+yY7r/05/ja+A6tfdt4PaQ0xiZWFKrK3s9Zl1B++2dDO5VByzLIhwy+BFKVswLe2fg9gbzV9st11pA5Avtu+QlUPAIjIImAosLR8oGt6WIAzBehxx7zjGhVWiYgXiFbVXKB8Sm+1iOzAyaqy3Zhqiq/8XLOB2eBYws/wczACSW0df6tfgf+/vbuPubKu4zj+/gAKifmQT0mEkGCTVFTUNEWYzjJt2YMlmBO3Zum0rS160B6mtFk+LFu2aTmZzTadWRqlhk40zEiwIfJgKiD4gFOZ0CQV4fbbH7/fLRc3T+c+51znXOe+P6/tjHPuc13X+X5h9/3l+7t+9+/3+I3p/cF7pntSxaE/z/hrrk0b0r2hnt3QXiNg3ORUhEZOcDdkWymzKM0HxkgaBbwMTAbO7XHMTGAqMBc4G5gdEd3Ddt+TtBvwLjARuF7S7sAHI+KVXMTOIM3AA7iH1DU9IukQ0v2mNZL2A96IiK7cOY0BVkTEG5LelHQ88DhwPnBDSX8X1ko1z/i7YfOMv6H7b1mkPOOv99a9UFhF4e/uhqwupRWlfI/oUmAWaUr4jIhYImk68EREzARuAW6TtIzUIU3O566V9AtSYQvgvoi4V9IBwMw8lXwgMBu4KX/kDGCGpMWkQjY1F7iTgemSNgFdwEUR8UY+52I2Twm/H09y6Lu2OePvHXh18ZaFyjP+alfshp57ENbk37pwN2QN8IoOveQVHfq4bc34W1fY1bc442/YUanI9acZfzvqhsZ8Ok1S2HeMuyHbild0MKtHPTP+9h+75dDffof2nRl/mzbAqn+mIuRuyFrAnVIvuVOyna7x1+kz/rbbDZ24eU05d0PWS97kryQuSrZNO9vVt8oz/nbUDXVvejdqAuw6tL1xWkfz8J1ZK213xt9/tixUVZnxt3ZVYRWFOVt2Q+OnuhuytnFRMivLwEHw4cPS4+jz09dqnfE37KhUqA48Mt3natR2u6GD4MgpeRUFd0PWfi5KZq20yxAYfkx6dNvpGn9jtuymap3xt71uaORJMP6CdH9on9HuhqxSXJTM2q2WGX/LZ8NTd6T3tjfjL7pg1WOpCC17ENY8m453N2QdxBMdeskTHawtes74W70gPd4pzPiTYONbMHAwjDwxT1JwN2TV4IkOZn3J9tb4W/v85kL1XheMPjUNz7kbsg7lomTWqQYMgH0OTo8jvtLuaMyaokN+m8/MzPoDFyUzM6sMFyUzM6sMFyUzM6sMFyUzM6sMFyUzM6sMFyUzM6sMFyUzM6sMLzPUS5JeB1a1O45e2hdY0+4gWsw59w/OuTOsAYiI03d2oItSPyDpiVrWnOpLnHP/4Jz7Hg/fmZlZZbgomZlZZbgo9Q+/bXcAbeCc+wfn3Mf4npKZmVWGOyUzM6sMF6UOJ+l0Sc9IWibpB9t4/3pJT+bHs5LWFd4bIekBSU9LWippZCtjr1eDOV8jaUnO+VdSZ2zJWkPOIyQ9LGmBpKcknVF477J83jOSPtPayOtXb86STpP0b0mL8p+ntD763mvk37jw/npJ01oXdQkiwo8OfQADgeXAx4BdgYXA2B0c/y1gRuH1I8Bp+fnuwG7tzqnMnIFPAY/lawwE5gKT2p1TM3Im3We4OD8fC6wsPF8IDAZG5esMbHdOJed8FDAsPz8MeLnd+ZSZb+H9PwJ/AKa1O59GHu6UOttxwLKIWBER7wJ3AGft4PgpwO0AksYCgyLiQYCIWB8Rb5UdcBPUnTMQwBDSN/1gYBfg1RJjbZZacg5gj/x8T2B1fn4WcEdEbIiI54Fl+XpVV3fOEbEgIrrzXwIMkTS4BTE3opF/YyR9AVhByrejuSh1to8ALxZev5S/thVJB5H+pzw7f+kQYJ2kP+XhgGslDSw12uaoO+eImAs8DLySH7Mi4ulSo22OWnK+AjhP0kvAfaQOsdZzq6iRnIu+DCyIiA1lBNlEdecraSjwfeDK8sMsn4tSZ9vW/ZDtTaecDNwVEV359SBgAjANOJY0bHBBswMsQd05SxoNHAoMJ33DnyLp5FKibK5acp4C3BoRw4EzgNskDajx3CpqJOd0AekTwNXAN0uLsnkayfdK4PqIWF9yjC3hotTZXgI+Wng9nEJL38NkNg9jdZ+7IA8XbALuAY4uJcrmaiTnLwL/ykOV64H7geNLibK5asn568Cd8H5HOIS0Rlpv/r6qpJGckTQcuBs4PyKWlx5t4xrJ95PANZJWAt8GLpd0adkBl8VFqbPNB8ZIGiVpV9IP4Zk9D5L0cWBv0o394rl7S9ovvz4FWFpyvM3QSM4vABMlDZK0CzAR6IThu1pyfgE4FUDSoaQfWK/n4yZLGixpFDAGmNeyyOtXd86S9gLuBS6LiMdaGHMj6s43IiZExMiIGAn8ErgqIn7dutCby0Wpg+UO51JgFumH650RsUTSdEmfLxw6hXSzOwrndpGG7h6StIg0fHBz66KvTyM5A3eRZjgtIs1uWhgRf2lR6HWrMefvABdKWkjqDi+IZAnpf9dLgb8BlxSGcCurkZzzeaOBHxd+NWD/NqRRswbz7VO8ooOZmVWGOyUzM6sMFyUzM6sMFyUzM6sMFyUzM6sMFyUzM6sMFyWzCpF0RRVWeZa0UtK+7Y7D+h8XJTMzqwwXJbOSSRoq6V5JCyUtlnROsRORdIykRwqnjJM0W9Jzki7MxxwoaU7+RdDFkibkr98o6QmlPaKuLHzmSklXSZqb3z9a0ixJyyVdlI+ZlK95t9J+WjcV144rXOs8SfPyZ/+mQxbutQ7lomRWvtOB1RExLiIOI62ssCNHAGcCJwA/kTQMOJe0qvmRwDjgyXzsDyPimHzORElHFK7zYkScADwK3AqcTVrrb3rhmONIKwUcDhwMfKkYSF7O5hzgxPzZXcDXepG7Wa8MancAZv3AIuA6SVcDf42IR7XjDW//HBFvA29LephUOOYDM/KaffdERHdR+qqkb5C+lw8kbf72VH6ve+20RcDuEfEm8Kakd/L6cADzImIFgKTbgZNIyzF1OxUYD8zPMX8AeK2uvwWzGrgomZUsIp6VNJ603cDPJD0AbGLzSMWQnqdsfYmYk7fZOJO0ZcG1pA5oGnBsRKyVdGuPa3XvIfRe4Xn36+7v/a0+q8drAb+LiMt2kqZZU3j4zqxkefjtrYj4PXAdaYuQlaQOBNJGdEVnSRoiaR9gEqlLOQh4LSJuBm7J19gD+B/wX0kHAJ+tI7zj8srUA0jDdP/o8f5DwNndC5pK+lCOxawU7pTMync4cK2k94CNwMWkYbBbJF0OPN7j+HmkrRdGAD+NiNWSpgLflbQRWE/aJ+h5SQtIW2CvAOrZpmEu8PMc4xzSHkTvi4ilkn4EPJAL10bgEmBVHZ9ltlNeJdysn5I0CZgWEZ9rdyxm3Tx8Z2ZmleFOyczMKsOdkpmZVYaLkpmZVYaLkpmZVYaLkpmZVYaLkpmZVYaLkpmZVcb/AW8Nru/gmitKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch3_2.best_score_, gsearch3_2.best_params_))\n",
    "test_means = gsearch3_2.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch3_2.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch3_2.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch3_2.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch3_2.cv_results_).to_csv('my_preds_subsampleh_colsample_bytree_1.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(colsample_bytree), len(subsample))\n",
    "train_scores = np.array(train_means).reshape(len(colsample_bytree), len(subsample))\n",
    "\n",
    "for i, value in enumerate(colsample_bytree):\n",
    "    pyplot.plot(subsample, -test_scores[i], label= 'test_colsample_bytree:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'subsample' )                                                                                                      \n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig( 'subsample_vs_colsample_bytree1.png' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
